Ask HN: Is anyone building automated long-term investing software?
Every now and then I see a post for automated HFT or similar trading bots, but these are usually research projects to pack a resume. This got me thinking: is anyone working on the opposite: automated long-term trading bots?
It should be designed to pick and hold a position for >12 months, use alternative data sources, and be open/hackable.
I guess it's time to tell the story online?
When I graduated college, I spent 3 months as a programmer with my econ friend trying to build exactly this. I started off creating a system to paper trade stocks retroactively. So you imagine you go back in time and pretend it's January 1st, 1982 then have an algorithm look at the stocks then, then move it a day forward, and let it trade for the past 40 years and see how it does.
We tried linear models, SVMs, neural networks, RNNs, ensembles, genetic algorithms, anything with stock data, news sentiment data, classic quant structures, and everything in-between. Basically, 3 solid months of coding before I started working.
Anyway, I found out a lot of stuff the hard way, because I didn't have an econ degree.
First off, you try enough methods, you end up p hacking or hill climbing the past anyway, and it's no good.
Second off, historical clean data is hard to get. It may or may not have splits in it or other things, so you may inadvertantly supply information from the future when playing back from the past. It's hard to get this right.
Third off, for many of the models we used, they were almost always competitive in the 80s (even a linear regression), but in the oughts or 2010's, they stopped being competitive. We thought computer based trading was becoming more competitive in hedge funds.
Fourth, simple models tended to work better. So for instance we may have trained the model on data from 70s-80s, then starting in the 80s, we did online (continuous) training as we moved the model forward in time. There's just not enough data. You can train on all historical stocks or all stocks or related data streams in the industry up to that point, but I think we probably didn't have enough data and the market is competitive.
Fifth, I wish I read a Random Walk Down Wall Street earlier, or all of Taleb's stuff. These are books that have deep mistrust of quants.
Sixth, I think to be competitive, you need to have money in the game, many heuristics, and industry experience. Big firms have this and equipment, but it's hard to get in as an individual.
Seventh, I put several hundred hours into this project and learned a bunch about machine learning and economics. In every way I loved the experience, and I'd encourage you to try it. Probably I'm a n00b here, but I hope some of my notes can help you.
I have this idea that we only have one universe of historical financial data, and it is only 500 years long, which is ridiculously small. So backtesting and drawing conclusions is highly overrated.
Another thing as you said is that it's hard to get quality data. For example most databases don't include price history for bankrupt companies (or miss quite a bunch), which makes some quantitative strategies like focusing on low PE and PB for example, completely bogus. Which is sad because most books will actually tell you to do that, without ever talking about how many of those backtests lack companies with -100% return in their virtual portfolios. Those tend to be low PE companies that the market consider risky, and it was right, but because they disappeared, the strategies outperform because they ignore so many losers.
To develop a trading EDGE. You're looking for a market inefficiency. The model is not dependent on perfect data or even accurate data. The model is even tested with having random prices... by using a FILTER to see if it still holds.
Then, you're going to paper trade it. Then live trade it. Historical data can only give you a directional indicator... is your 'thesis' of market inefficiency... directionally accurate.
I don't think we have 500 years of data. Anything before 1926 is pretty sketchy.
Indeed, I pulled 500 out of my hat because that's roughly stock market history. But that kind of reinforces my point, if we only have 100 years that's very little, especially considering how fast things change. Early 20th centuries very few industries were publicly traded, even amongst those that existed at the time. And even beyond the industries, other things have changed a lot like regulation, taxes, accounting rules, management style... surely the market takes all of those into account, one way or another.
Yeah, the thing that makes trading and investing different from most other disciplines is that the distributions are completely non-stationary and are changing all the time. There are some "stylized facts" (that's the term to search for) so use those to at least ground your model but you won't make any money from that.
Yeah if you want good data, start collecting it now. I believe anyway the magic is outside the numbers. They are only a shadow on a cave's wall.
I agree, there is a qualitative statement to be made. AI can help especially summarizing text like news and earnings calls, but there is quite a bit of human work to be made beyond just running some software.
Kudos for learning a lot in a short time. Many take longer to recapitulate these ideas, I certainly did. Alas, this is the standard buy-side quant curriculum. IDK where SOTA is these days but you have to a different edge because as you highlighted, it's so easy to reproduce the basics.
As a fun aside, I find it quite fascinating when you see a particular signal go dead overnight.
I recall one particular overnight reversion strategy in my local market which in the 90s and early 2000s had like a 45 degree straight line equity curve and then overnight in 2011 it just went flat. Someone clearly turned on a model that day and started trading it. It's a small market here so I know who it was but I still find it fascinating how clearly it showed up in the data.
I had a simple trading structure that I summarized from a few sources. Curious what you might think of it.
1. Filter all underdog stocks.
2. Have a catalyst detector - e.g. OpenAI announces new model. Link that to NVIDIA, MSFT, etc.
3. Among these stocks, when you see a marubozu break through the previous resistance point, buy the stock. Then sell it near peak. (Need a peak detection algorithm)
The trick is having all three work fine. But it's easier to debug and test when one doesn't work if you break it into parts. You also don't need one really good methodology, you just need a few decent ones.
Or you could find similar patterns that can be broken down into small parts and do those too.
> Need a peak detection algorithm
I think thats all you need. ;)
In realiity, trading is hard because it requires 2 points where you have to get the timing right.
First you have buy the position at a time when it is favorable. Then you have to exit the position when you get profit. Sounds easy, but the hard part is that this profit also has to cover any past losses. where you failed timing the entry into the position.
Its easy when the stock goes up 800% and then falls 50% to 400% your original position, that you should have sold when it was at its "peak". But along the way to 800% you had so many times to sell for 500%, 600%, 700%, etc. and along the way the stock had fluctuations with many peaks.
If you sell to early you can't get enough profit to cover past and future losses. If you sell to late same story. So you have to nail the exit position also and that is where most models that rely on past data fail. People just walk through the parameters until the entry and exit positions on their test data line up to make a profit, but then can't replicate when going "live".
Another way to look at investing is everyday you are in the market, it is almost the same logically as selling and choosing to reinvest every day. So if you hold AAPL for 5 years, that is about 1000 days where the algo is choosing to invest (i.e. keep invested) in AAPL. Its pretty tough to have that many decisions points because even 0.1% noise would cause you to sell.
What we did was compare it to the previous peak, and then 1.1x that or so. That works well for trading breakthrough patterns.
If you're going long term, then it's quite different because there may never really be a peak. There might be corrections or something every now and then, but they whole idea behind long term is that these don't matter.
I would actually tie it to the other two - you'd have to detect when a stock is overpriced, and you'd have to detect catalysts.
Still tough. Meta sunk on their rebranding not so much because of the metaverse but because FB had been dead for a while. But the announcement was the catalyst. If Meta had been underpriced, it would have been a positive catalyst and people would have applauded the metaverse. Which would probably have triggered catalysts on RBLX, MSFT, and other metaverse players.
NVDA had been underpriced for a while, and the multiple catalysts have made it shoot up, though it's possible another one in the future could make it crash.
So IMO it's not just one point, the other detectors also help to filter the noise.
Did you ever put anything “into production” and actually start using it? And if so, are you still using it?
No. I trade manually based on fundamentals, diversification, and limited downside options. I've lost interest and respect for quantitative strategies while I've worked for and run my own businesses. So consider me a failure here and this is only a story of how not to build a lightbulb.
I wrote a scraper that downloads 10-Ks from the SEC and then built a simple analysis based on high dividends/stock buybacks and a discounted cash flows analysis. It found some pretty undervalued companies that ended up doing fairly well over the last year, but not as well as the tech stocks.
As others noted the datasets are not really standardized even with the SEC Edgar data so there is a lot of massaging you have to do.
How automated was the DCF? The math is simple but requires many assumption at multiple steps
It was completely automated. And you are right I made some assumptions.
Long term investing means buy and hold. And hold and hold and hold.
A system that does that for you would not really be a trading bot per se, it would just be a general algorithm for "picking stocks". Automating the actual purchasing is probably unnecessary.
If you find one, let us know! Most investors in the world are searching for the such a thing.
And the easiest of such algorithms - ala, just buy a bit of everything - seems to be the winner imho. AKA, indexc investing.
And luckily, this involves roughly zero time and effort. And carries little to no risk of fucking it up enough that you lose everything (see: margin, selling short, day trading, crypto).
https://www.bogleheads.org/wiki/Target_date_funds
Seriously. Target funds if you're lazy or a three fund portfolio you rebalance every couple of years. Don't need software for that
Shades of Parkinson's Law [1], along with the idea that more effort is more valuable than less effort. Sometimes, simplicity is the ultimate sophistication.
I bet we would could reduce US capital market trading to four days a week and still arrive at a similar outcome wrt efficient price discovery and liquidity windows, tangentially [2].
[1] https://en.wikipedia.org/wiki/Parkinson%27s_law
[2] https://www.bloomberg.com/opinion/articles/2021-10-06/why-no... | https://archive.today/Ykapv
In the long term stocks and software are smoke and tissues.
Actual long term investors today are looking at an additional two billion people by 2050, increased demand for food and water, and regional destability due to climate change.
Long term investors today are buying land and resource access about the globe, or moving to secure such things via private contractor | mercanary armies.
China has purchased one in four US pigs (the farms, the feed, the processing), the Saudis have locked in access to large quantities of US aquifers, and Eric Prince wants the US to retake Africa: https://theintercept.com/2024/02/10/erik-prince-off-leash-im...
These are all examples of securing access to water and food resources to ensure supply into the long term.
The investment payoff of is having those resources when others don't, being secure in what you need and being able to profit from what you don't in times of extreme demand.
The vast majority of my bet would be on climate change. People aren't acting on it fast enough to turn it. It's been modeled very well, so you know what will happen in 2050 if things don't change. It will cause an insane amount of damage, especially in archipelagos like where I live. You can't just add more drainage and try to drain the sea. New York and Singapore are raising their shorelines, but most places just don't have this kind of terraforming capacity.
Property is the easy one, but it's useful to look at things like farms along affected areas.
> so you know what will happen in 2050 if things don't change.
Sure, Siberia will be open praire cattle country; cossacks and reindeer herders will be cowboys mustering steers.
Russia knows this also, hence the hiring of US cowboys to train russians and the recent prevalance of Russian Rodeos in both the US and in Russia.
This all goes to the question of long term investment, projected affected areas are being abandoned by deep investors who have turned to probable new prospects.
Farming real estate questions here in Western Australia have already turned to what land will be prime in 20 years and what used to wet boggy questionable land is already fetching a climbing price in anticipation of when it becomes "sweet".
Yeah "long term investment" is called wealth management. It's a world apart from algorithm based or retail stock trading.
Indeed, the software I worked on here in my home state back when I was interested in long term investment is still about today, 14+ years back it was broken into parts and sold on, here's one section, now run more or less "as was" by S&P
https://www.spglobal.com/marketintelligence/en/campaigns/met...
Our clients way back when were mostly all billionaires, some personal, some corporate; between client discussions, GIS data linking, trends, etc we were able to get a line on a bunch of penny stocks that were selling then ( ~2003 ) for 2c (AU) and now trade at $20+ each.
None of that came from looking at day trading, it came from deeply looking at global energy and resource demands against known and suspected sources as yet unexploited.
My automated long term investing bot is a standing order to transfer money each month from my cash account to a whole market index tracker. Happy to compare my returns to yours after a decade or few.
> automated long-term trading bots
You don't need trading bots for long term investing or even infrequent trading. In LT investing, portfolio tracking and asset allocation/reallocation are the primary tasks. Robo-advisors were very popular almost a decade ago. Most brokerages have integrated such features now. Also, checkout M1 Finance.
I started investing first with the help of spreadsheet then shell scripting and now Jupyter Notebooks and Python. Beyond LT investing portfolio tracking, majority of time I spend on short to mid-term strategy development, back-testing and implementation; portfolio hedging and leverage; and options trading.
Only manual aspect is actual order placement, which takes only few minutes at best.
With most trading platforms. The api is available to do the trades. Obviously, look on github.com for projects, tie-ing into the trade platform of choice.
This platform, allows one to do automated trading based on your own strategy. US only traders, for now. https://www.composer.trade
If you are just doing portfolio re-balancing. Say, twice a year. You could re-balance based on each stock's risk parity.
i.e. Risk parity is an approach to investment portfolio management which focuses on the allocation of risk, rather than the allocation of capital. The risk parity approach asserts that when asset allocations are adjusted to have the same level of risk, the portfolio can achieve a higher risk-adjusted return.
Some Quant Resources: https://quantpedia.com
They teach a class on quant. Pretty good. Python oriented. https://quantscience.io
I'm using LLMs to basically build "junior analysts" that monitor very niche types of companies -- think, junior mining companies, or very specific commodities futures... A lot of these spaces have tons of terrible companies and there's a lot of noise, so if you use a framework that is concrete enough, you can have LLM agents do various types of research for you, fill in the framework, and sift through the noise for you.
Case in point, my framework for mining companies is here: https://emergingtrajectories.com/a/pub/mining_company_risk_f... You can see the scores here: https://emergingtrajectories.com/c/copper_mining_companies
"Long term" -- we'll see, I expect to hold positions for 12-24 months.
For those interested, my work above is influenced by two important books: "You Can Be a Stock Market Genius Even if You're Not Too Smart" by Joel Greenblatt and "Superforecasting: The Art and Science of Prediction" by Philip Tetlock. The idea from Joel's writing is to look for less liquid or less popular asset classes (or ones that structurally can't be invested in by the pros who are smarter/better-resourced than you), and Tetlock really drills process and research for long-term forecasting.
Don’t people usually just buy etfs or funds for this sort of thing?
That’s what I do: VTSAX and chill. I set it and forget it and it has performed very well for me over the years. Also worth noting most institutional investors struggle to outperform this fund over extended periods of time
I’m looking for something a bit more specialised, ETFs being the absolute far end of automated.
Schwab has "slices" that act like an ETF that you decide the composition of
Where do you get the data for such a thing? Last time I looked into this (not to trade per se, but to see if I can come up with anything clever using machine learning) the complexity of the data landscape made my head hurt. As did the cost for some of the more comprehensive options.
There are sites like WealthFront. It's designed to act as an automated fund manager, from what I understand. Buying and selling as needed to take advantage of things like tax loss harvesting.
I haven't looked at it in a while, but it was promoted heavily on some podcasts I listened to years ago when it came out.
I don’t live in America so WealthFront and other robo-advisors aren’t available to me
If you're willing to go to a robo just look at the funds/prospectus pages of robofunds, look at the holdings and rebalance every so often as needed
In Indian markets, I think Shyam does this https://stockviz.biz/
Edit:
Ah, I just realised you might have means software part more than the financial part. Shyam does publish R code of various things on GitHub
There is no reason to make a long-term trading bot because that's what ETFs and various derivatives already are. They are algorithmic assets that you can customize according to your risk preferences, and you can just buy them.
If you actually succeeded at this, would you ever tell anyone?
Why?
My theory on this is: algo trading on short term, if it works you don’t tell anyone because by trading on the opportunity they decrease the available opportunity to you (efficient market, chaotic systems etc).
Long-term trading doesn’t work like this, in fact you want more people to identify the opportunity and buy to improve your position in the long term, but generally you should be investing on alternative signals not technical analysis.
In other words --- what you and I think isn't nearly as significant as what others are persuaded to think.
For example, I personally think TSLA is way, way, way over bought. The reasons for this are many and varied but still largely irrelevant. My opinion didn't stop an "analyst" from issuing a buy recommendation yesterday with a target price of $265.
Maybe this "analyst" has clear, logical reasoning with supporting computer analysis --- or maybe his grandma is just heavily bought into TSLA. In any case, the stock jumped 2.5% yesterday.
Bottom line --- everybody and his brother is using software. But opinion and psychology remain significant factors that defy computerization. The market can still stay irrational longer than you can stay solvent.
For one, if you wanted to make $20k per month, you'd need at least $200k invested. Oh wait, we're talking long term, not trading. In that case, you'll make about $20k per year. So you'd want $240k per year to pay that kind of salary, meaning you need around $2.4m in the bank.
It's a lot easier to raise that money by selling the tool than it would by trading the stocks.
It also doesn't hurt to share knowledge. Very often you can compare your notes to another.
In fact, if you had an algorithm that picked winning stocks, just add a little delay to everyone else's bots. Your bot buys it first, then some whale buys it and pumps your stock.
>This got me thinking: is anyone working on the opposite: automated long-term trading bots?
What would be the point? HFT works because you can beat the market by being faster, I don't see how long term trading could beat the market unless you have insider information.
And if you can't beat the market, there is absolutely no point in the bot, as you can trivially just buy an index fund tracking the market. Which is also what I am doing, I would never use a bot over that, as it is just additional risk.
I think brokerages have this kind of things ? I saw ads from Revolut recently with something similar.
You are talking about 2 different things in your post though, I believe: 1 - automating long term investmenets (this is the Revolut thing) - ie, setup an amount you set for investment every month - and it automatically buys whatever you want 2 - a research tool ? (not a bot though)
Or, it just hit me while writing, are you talking about quantitatives strategies ?? If yes, then yeah, half of Wall Street was working on that ! There were some open source attempts, I think the best known was Quantopian - look it up.