"Invest in projects, not papers." is the typical thing you hear from late-stage researchers that forgot what it's like to be a PhD student or early post-doc. We have to publish as much as we can, or our supervisors and committees won't allow us to progress in our careers.
I can't spend a year on topics that seem interesting but that might not yield papers if they don't work. From the bureaucratic point of view, which is almost all that matter for junior researchers, that would be simply time in the bin.
I would love to spend years on something I care about without caring how many papers it will generate, but if I do that, I won't have a career.
I actually did invest in projects rather than papers and I definitely feel I paid the price. It's the main reason I opted for a postdoc rather than going straight on the academic market: the quality of my research was high, but the number of publications I have is too low to be competitive. At least that's what my PhD advisor said and I honestly agree, especially after speaking with people who just landed tenure-track jobs and more senior professors.
What I have to say may come across as harsh but I only intend it to be helpful.
If you invested in projects, and the investments did not pay off, I see three likely explanations.
First, maybe your investment function is miscalibrated.
Second, maybe your investment function is well calibrated but your time horizon is too long.
Third, maybe you are unlucky.
I have no insights into what happened in your case, I don't even know what field your are in.
But, case 1 suggests you may be unsuccessful in academia.
Case 2 suggests possible success after adjustment to more immediate reward.
Case three suggests possible success after a difficult recovery process and further investments.
None are ideal paths but cases 2 and 3 suggest possible recovery strategies if you are deeply committed to academia. The optimality of such an approach is highly subjective.
You likely should not feel like academia is the only available path, if you do that is a red flag that something is amiss.
Even then, there are too many PhDs for few spots in academia and top-tier labs, so there's no way a PhD dropout could compete unless they have some other kind of experience.
Also, it's not that hard to publish in high-end NLP conferences, so it doesn't say much.
I watched a startup give an NLP AI engineer with only 1 workshop paper at ACL get a 150K remote offer less than a month ago. This person is in the middle of their BS program right now.
It is most certainly not easy to publish top AI research (unless you are doing unethical things). I repeat, if you have a NeurIPS main conference publication, you don't need to have a degree for a top AI lab to at least consider you.
If your experiences are different, my guess is that you're not an American.
No, not an American. Did you mean that Americans have lower standards than Europeans in this respect? I'm confused about your last sentence.
Anyway, my note was specifically about publishing NLP research at ACL conferences (ACL, NAACL, EMNLP, etc.), where publishing mediocre work hasn't been difficult. Just go through the proceedings, and you'll see that's the case. I don't know about NeurIPS.
This has good advice about academic research in general not just AI Research. It's huge to "Select timely problems with large headroom and "fanout"" - you need a research program to have a successful research career. At least most people do. It's also huge to "Put your work out there and own popularizing your ideas" - the second part is hard for a lot of academics to do. Putting your work out there is huge to get feedback, which may help you pivot or try different directions, as well as fostering collaboration. Collaboration is a large part of being a successful researcher.
> Collaboration is a large part of being a successful researcher.
I would even consider it as the largest part. It brings ideas, funding, spreading of your ideas, access to institutional research grants/projects, industry contacts, and more.
Agreed! It also brings friendships, new ways of thinking and framing, inspiration, and the opportunity to give and receive support - to discover how much you can matter, and the good you can do.
The idea is that rather than doing a bunch of completely independent research projects, all of your projects are designed in service of answering some larger research question.
At least some of it comes from "hype" too. The author of Dspy (the writer) (https://github.com/stanfordnlp/dspy) should know this, given that Dspy is nothing more than fancy prompts optimizing prompts to be fancier according to prompt chains described in papers (i.e. Chain of thought, Tree of thought, etc). Textgrad (https://github.com/zou-group/textgrad) is an even worse example of this, as it makes people think that it's not just a prompt optimizing another prompt
Dspy has 17k stars, meanwhile PyReft (https://github.com/stanfordnlp/pyreft) isn't even at 1200 yet and it has Christopher Manning (head of AI at stanford) working on it (see their paper: https://arxiv.org/abs/2404.03592). Sometimes what the world deems "impactful" in the short-medium term is wrong. Think long term. PyReft is likely the beginning of an explosion in demand for ultra parameter efficient techniques, while Dspy will likely fade into obscurity over time.
I also know that the folks writing better samplers/optimizers for LLMs get almost no love/credit relative to the outsized impact they have on the field. A new sampler potentially improves EVERY LLM EVER! Folks like Clara Meister or the authors of the min_p paper preprint have had far larger impacts on the field than their citation counts might suggest, based on the fact that typiciality or min_p sampling is now considered generally superior to top_p/top_k (OpenAI, Anthropic, Gemini, et al still use top_p/top_k) and min_p/typicality are implemented by every open source LLM inference engine (i.e. huggingface, vllm, sglang, etc)
I think that's a little harsh. Imo the reason for difference in popularity/github-stars is just different user bases- an order of magnitude more people use LLM APIs (and can leverage Dspy) vs those who finetune an LLM.
Agree about the abstractions btw. I found Dspy very convoluted for what it does, couldn't make sense of Textgrad at all.
I like the article directionally but fear the examples are too short-sighted for most AI researchers.
Picking something useful in 1-2 years is a reason to go to industry, not research, and leads to mostly incremental units that if you don't do, someone else will. Yes, hot topics are good because they signal a time of fertile innovation. But not if your vision is so shallow that you will have half of IBM, Google, and YC competing with you before you start or by the time of your first publication (6-12mo). If you are a top student, well-educated already, with top resources and your own mentees, and your advisor is an industry leader who already knows where your work will go, maybe go to the thickest 1-2 year out AI VC fest, but that's not most PhD students.
A 'practical' area would be obvious to everyone in 5 years, but winnowing out the crowd, there should not be much point to it today nor 1-2 years without something fundamentally changing. It should be tangible enough to be relevant and enticing, but too expensive for whatever reasons. More fundamental research would be even more years out. This gives you a year or two to dig into the problem, and another year or two to build out fundamental solutions, and then a couple years of cranking. From there, rinse-and-repeat via your own career or those of your future students.
Some of my favorite work took 1-2 years of research to establish the problem, not just the solutions. Two of the projects here were weird at first as problems on a longer time scale, but ended up as part of $20M grant and software many folks here use & love, and another, a 10 year test of time award. (And another, arguably a $100M+ division at Nvidia). In contrast, most of my topic-of-the-year stuff didn't matter and was interchangeable with work by others.
I find that in AI / ML / computer vision, the thinking horizon has shrunk a lot. Reviewers prize straightforward incremental work that nevertheless beats the SOTA. New problem formulations are too hard to decipher and understand deeply and nobody has time nowadays. Everything is super sped up. There's a fire hose of papers coming out that all do slightly different things, or the same concurrently.
In this area even the best, most respected professors don't have long term laid out research plans. Grant plans must exist but they never quite pan out that way. Researchers in this environment are very reactive and it's all about being fast. Diffusion is popular? Then let's do diffusion for [my specialty], or transformers for [X]. Or combine LLM with [my last topic]. People are constantly pivoting and jumping from one opportunity to the next. In my experience gone are the days where you could carve out a niche and work in peace on it for 4-5 years and have a consistent overarching story in your PhD.
IMO that kind of thinking flips correlation/causation, and feels like some sort salience bias towards voluminous but crappy work. Yes, you can do it, just good bye impact. Worse, it's a self-fulfilling prophecy: by moving your work to be more incremental, you're even more at risk by the incremental advances of others.
The field exploded, so yes, there are way more good and even more bad papers, so some researchers interpret that as the need to publish a lot. Top labs attract a lot of students and funding, so the PI spends little time per student, and instead farms out to post docs who are there for only a few years, and in turn they do most of the phd student mentoring. And many folks want to bean count neurips papers. The result is yes, if you're unintentional about impact, and get distracted by the volume, you too can optimize for that, do lets of incremental work, ultimately you've learned some skills and wrote papers that won't matter. Google/FB/OpenAI/Spotify/LinkedIn/etc will still hire you at the end, that's fine, they hire a ton of phds.
Objectively, there are some real jumps happening, but zoomed out, very few of the papers coming out in a given year change all that much. E.g., we're using LLMs for some hard societal problems (misinfo, cyber, ...), but nothing fundamental changed in our startup's work since gpt-4 came out, and all versions before gpt-4 weren't significant enough to motivate using generative LLM methods. There are a few methods we're tracking (e.g., graph RAG, certain kinds of agents), but most work is embarrassingly shallow or bad, so the incremental papers have done almost nothing to influence us, and we learn more from other industry folks doing more ambitious or real work here. The previous disruption on our AI side was around GNNs, but that was over many years and quite rocky. So if we were doing all this as a serious PhD project (invention) instead of as a startup (innovation), I'd expect similarly low levels of disruption from the perceived fast pace of others.
True, but at the end of the day, a mediocre PhD that gets you a nice job is, all things considered, not a bad outcome from the point of view of the student. Yes, obviously becoming MrBeast is better than being the (n+1)th gaming streamer. But I think some might not realize how many PhDs are churned out constantly. PhDs are also given out by mediocre universities, advised by mediocre profs. Not everyone can be the next AI superstar.
Beware the extreme selection/survivorship bias of listening to advice from the top most successful researchers and household names / celebrities.
Most PhDs make a small dent in a small niche specialty, get their degree and go on with their lives, and their work is superseded 6 months to 2 years later.
As I wrote, different folks will already be chasing more immediate projects as they're too attractive: Think masters students, industry labs, startups, etc.
Rules are meant to be broken. For example, a professor will often put a masters student on a cute me-too one-off project in a hot area as the short-termism helps the student have a smoother time, and the professor is trading expected low impact in the the project for a cheap shot to see if there is a more interesting question behind the immediate one.
Much of the PhD is about learning to spot & navigate problems, not solutions. Young PhD students can totally pick the same me-too problems of masters students. The advisors would likely just think the student is using the same approach to seeing where the PhD worthy problem is. If still nothing, should be ready to drop the topic, or if they truly love me-too work more than the PhD process, leave for a better environment for that, like a company / industry lab / startup throwing more resources at the problem.
Also, not every phd needs to do the original topic of attempting impact. You can totally use the PhD years to learn how to do me-too & unimpactful work or even the most weird & niche problems: as long as at least one advisor thinks they can stay funded while supporting you, most programs will let you through.
You're right, but the individual has to ask if entering a crowded niche is worth it or not. Does your main strength lie in absurd workload to grind out the paper a few months/weeks earlier to Arxiv than the next group will?
The risk reward calculus still often benefits this approach. The risk is that you get scooped. If you do something nonobvious, the risk is that it doesn't pan out or the reviewers don't get the point. The latter is harder to solve by grinding. So many phds will chose the former, as at least putting in ridiculous hours feels more actionable. Also, doing the straightforward thing and getting scooped is less ego-hurting than failing at your own special pet idea.
I think many who aren't in grad school don't understand how extremely important it is to have publications. At the beginning you must get publications to do internships and to build contacts at conferences. Then later it becomes imperative in order to graduate and get your next position. Nowadays you must even have multiple top conference papers just to start a PhD.
Of course from the point of view of the advisor with 15 PhD students, the ideal student would do high risk high reward work and if 3 projects become super successful, 5 become quite successful and 7 yield nothing, that's still great for the advisor. But the 7 who lost several years would rather not go through this.
... and when everybody decides to be a contrarian, the contrarian position is to do the popular thing. But the catch is that this never happens on practice.
Unfortunately while this advice sounds useful, it isn’t. It might be useful if you measure impact is the fake metrics like citation count but if you want to measure with actual tangible impact on the real world, you have a rude awakening before you. To me there are 2 ways a paper can create impact:
1. The paper represents such an impressive leap in performance over existing methods in AI, that it is obviously impactful. Unfortunately, this way of generating impact is dominated by industry. No one can expect Academia to train O1, SAM, GPT5 etc. AI rewards scale, scale requires money, resources and manpower and Academia has none. In the early days of AI, there were rare moments when this was possible, AlexNet, Adam, Transformers, PPO etc. Is it still possible? I do not know, I have not seen anything in the last 3 years and I’m not optimistic many such opportunities are left. Even validating your idea tends to require the scale of industry.
2. The paper affects the thought process of other AI researchers and thus you are indirectly impactful if any of them cause big leaps in AI performance. Unfortunately here is where Academia has shot itself in the foot by generating so many damn papers every year (>10,000). There are just so many, that the effect of any 1 paper is meaningless. In fact the only way to be impactful now is to be in a social circle of great researchers, so that you know your social circle will read your paper and later if any of them make big performance improvements, you can believe that you played a small role in it. I have spoken to a lot of ML researchers, and they told me they choose papers to read just based on people and research groups they know. Even being a NeurIPS spotlight paper, means less than 10% of researchers will read your paper, maybe it will go to 50% if it’s a NEURIPS best paper but even that I doubt. How many researchers remember last year’s NEURIPS best paper?
The only solution to problem 2, is radical. The ML community needs to come together and limit the number of papers it wide releases. Let us say it came out and said that yearly only 20 curated papers will be widely published. Then you can bet most of the ML community will read all 20 of those papers and engage with it deeply as they will be capable of spending more than a day at least thinking about the paper. Of course you can still publish on arxiv, share with friends etc but unless such a dramatic cutdown is made I don’t see how you can be an actually impactful AI researcher in Academia when option 1 is too expensive and option 2 is made impossible.
> Academia has shot itself in the foot by generating so many damn papers every year (>10,000).
Each PhD student needs 3-5 papers to graduate. Nowadays you even need 1-2 to simply get into a PhD program. The huge number of papers compared to, say, 10 or 15 years ago simply reflects how many more PhD students there are in this area.
It definitely isn't sustainable but academia is a distributed system. There's no way to coordinate publishing less. No individual agent will rationally choose that. Especially that it's not just competition within AI but other CS fields too for huge grants and the grant agencies defer to citation and publication metrics. CVPR is among the top scientific publication venues by various metrics and this state of affairs also benefits the field.
There are reasons for why things are this way but the way it's going is clearly unsustainable. It's a red queen race, similar to grade inflation, credential inflation, and various forms of cost disease.
With one outlier, by the one academic AI lab that continues to produce impactful research every year, Christopher Ré's at Stanford (although the raised VC money now so who knows), FlashAttention.
If you were a really good academic, you could come up with a theory that predicts the performance of a model you haven't actually tested. Physics is full of this - theories that predict something novel followed by other people doing expensive experiments to test them. The guy who published the theory gets the credit. That's probably what we want from academic AI researchers - theories that we can eventually use to design models, rather than just haphazardly building something and wondering how good it will be.
That theory sounds great. There's no such thing at the moment and maybe never. Lots of smart people have tried.
Just because that's what people want, doesn't mean we can produce it. I often talk to funding agencies about things like this. "We don't want to fund boring research, only what will give us the ultimate theory of how everything works". That's not how science or progress work.
> That theory sounds great. There's no such thing at the moment and maybe never. Lots of smart people have tried.
In my opinion the problem rather is that considering the current AI gold rush, people are rather eager to throw newly implemented models around instead of thinking really deeply how an insanely better model could look like.
Article's point seems to be that in the long term: paper count, citations, impact, motivation and fulfillment will all come from focusing on a project.
Idk, nothing wrong with going for a low-hanging fruit and doing a one-off sometimes. So many academics fail to get stuff over the finish line. Not the right advice for everybody.
Depends on the person. Not unheard of to see people with 10+ working papers that realistically will never see the light of day. Each of those are months of work.
This article sort of lists the ideals, but is factually contradicted by the immense amounts of non-impactful and trivial AI papers, who have around 20 authors each.
There does seem to be a strong incentive to publish whatever and distribute the credit among dozens of people.
For the rare actually impactful research the advice is a bit trivial, you might as well quote Feynman:
1) Sit down.
2) Think hard.
3) Write down the solution.
> First, the problem must be timely. You can define this in many ways, but one strategy that works well in AI is to seek a problem space that will be 'hot' in 2-3 years but hasn't nearly become mainstream yet.
I think about research as, what can I bring that's unique? What can I work on that won't be popular or won't exist unless I do it?
If it's clearly going to become popular than other people will do it. So why do I need to? I'm useless.
Yes. You'll increase your citation count with that plan. But if you're going to do what other people are doing go to industry and make money. It seems crazy to me to give up half a million dollars a year to do something obvious and boring a few months before someone else would do it.
Artificial intelligence is becoming part of everyday life, and Apple’s advancements are a great example. Submagic https://ai-depot.net/tools/submagic/ is one tool that stands out for automatically transcribing speech to text, making content more accessible. It’s especially helpful for video creators looking to reach wider audiences across different languages. The tool handles transcription while you can focus on the creative aspects. It’s a time-saver that can enhance content quality with little extra effort.
Academic research in the USA today is entirely corporatized. Try publishing a paper on fine-tuning LLMs with the goal of creating agents capable of replacing the executive suite and the board of directors for major corporations and see what happens to your career. It's far worse today than what Eisenhower described c. 1960 in his final address to the nation, because so much of the research money is now doled out by corporate entities:
> "Today, the solitary inventor, tinkering in his shop, has been overshadowed by task forces of scientists in laboratories and testing fields. In the same fashion, the free university, historically the fountainhead of free ideas and scientific discovery, has experienced a revolution in the conduct of research. Partly because of the huge costs involved, a government contract becomes virtually a substitute for intellectual curiosity. For every old blackboard there are now hundreds of new electronic computers. The prospect of domination of the nation's scholars by Federal employment, project allocations, and the power of money is ever present — and is gravely to be regarded."
Unless you are independently (very) wealthy, you'll have to align your research with the goals of the funding entity, be that the corporate sector or a government entirely controlled by the corporate sector. You may find something useful and interesting to do within these constraints - but academic freedom is a myth under this system.
> Try publishing a paper on fine-tuning LLMs with the goal of creating agents capable of replacing the executive suite and the board of directors for major corporations and see what happens to your career.
I expect if you manage to get some interesting results, your career will be great.
The trouble is that there aren't any "official" benchmarks for evaluating performance for such a model. That's the main problem and makes it very hard to show if your models are any good.
A lot to like here, a couple of things to comment on:
> "If you tell people that their systems could be 1.5x faster or 5% more effective, that's rarely going to beat inertia. In my view, you need to find problems where there's non-zero hope that you'll make things, say, 20x faster or 30% more effective, at least after years of work."
This works for up-and-coming fields, but once something is stable and works at large scale, it's all about the small improvements. Making petrol engines 1% more fuel-efficient would be massive. Increasing the conversion rate of online ads by 1% could make you very, very rich indeed. Good advice for AI probably; bad advice in other fields.
> "Invest in projects, not papers"
The best way I think you can go about this is allocate some fraction alpha of your time to projects, and (1-alpha) to things that produce short-term papers. Alpha should never be zero if you want a career, but it will start out small as you begin your PhD and gradually grow, if you can make it in academia. At some point you'll reach a compounding return where the projects themselves are spawning papers - one way to do this is to get to the point where you can hire your own PhD students, but there are several others.
As long as your 2-years-into-a-PhD review as some unis have them is about how many papers you've published (somehow weighted by journal/conference rank) and how many others are in the pipeline, you need to focus on papers until the point when your institution will let you do something more useful. Think of it as paper writing bootcamp so that once you do get more time for projects, you'll have practiced how to write up your results.
> "Make your release usable, useful ..."
This is excellent advice, also for anything else related to code.
> Increasing the conversion rate of online ads by 1% could make you very, very rich indeed.
I claim there is actually a rather easy way to do this (but it won't make you rich!).
The basic idea is rather simple: for each potential (ad/product, user) pair, ideally store two kinds of information:
- the estimated probability that the user will click on the ad and/or make a buying decision
- the confidence that you have in the correctness of this estimate
Then only show ads where, based on this data, the conversion rate will be high with a high probability.
Result: by only showing ads to those few people who very likely want to buy your product, you create an insanely high conversion rate, but you loose money because the ad will be shown to rather few people (possibly these are even people who don't actually have to get "convinced" to buy your product).
In other words: we just created an artificial example of Goodhart's law ("When a measure becomes a target, it ceases to be a good measure") in action. What this lesson tells us is
- the "conversion rate" of an online ad is just a proxy for some entirely different business goal that you have
- the conversion rate can rather easily be manipulated
> Making petrol engines 1% more fuel-efficient would be massive. Increasing the conversion rate of online ads by 1% could make you very, very rich indeed. Good advice for AI probably; bad advice in other fields.
It's good advice in every field. Most improvements have a cost. If you make engines 1% more efficient but 5% more expensive no one will care.
Heck electric engines are wildly more efficient (80-95%) compared to combustion engines (20-35%), not to mention far simpler, and we can hardly get people to switch. Even immense improvements can be hard to roll out.
Don't work on marginal gains. Any minor problem or inconvenience will wipe them out and all your time and effort will be worth nothing. Let people who are paid by companies do that.
If you're going to do a PhD focus on something that could be big. Whenever students suggest a project I always start by asking what the upside is: what if we massively succeed?
Is this article roughly the application of the startup mindset to research? With the major downsides that you're giving away your insights for free, and potentially actively help other companies profit from your work. You don't go as far as to productize the research, but just make it really easy for others to do so by building "artifacts" for it.
I'm not a researcher, but have thought about doing a PhD in the past.
It's probably a lot more nuanced that this. Show progress but don't make it easily accessible. *Hide* something important for yourself. Kind of like modern day "open source".
This used to be very common but nowadays it's more and more expected to release the code with the paper. This makes it harder to obfuscate big parts of the method.
Arguably, this also has lowered the bar for doing incremental research. Just get the code of the sota from github, implement a new feature, test it, if it's 1% better then publish it.
Previously there was a natural gatekeep ing effect that you had to be able to understand and translate to code all the math and equations in the paper and be familiar with the dark knowledge details that everyone in the field knows but isn't spelled out explicitly in the papers. Boring bits like how to exactly preprocess, normalize, clean the data, how to precisely do the evals etc.
> Is this article roughly the application of the startup mindset to research?
Somewhat but a few crucial points are missing for it to be that. For example,
- hire a lot more people (PhD candidates, postdocs) than you have funding to pay till they finish. Keep growing and getting more grants until you get a Nobel prize (or equivalent) or go broke
- fake your data and publish in the top journals/conferences
- don't do any research yourself, just get others to do everything and "coordinate" them. Make sure you get all the credit for the publication. The way to get others to do things is to "collaborate" with them or maybe (promise to) pay them.
I would say follow the money, create a product or service that generates recurrent income and channel that income to study the fundamental problems that would interest you. Owning a product or platform opens up for a lot of opportunities.
I don’t think you would loose access to academia, actually it is the opposite. Academia would invite you with open arms once you have funds and/or the data :)
> Academia would invite you with open arms once you have funds and/or the data :)
This way of getting access to academia is based on the premise that you already did quite some research - which is much harder if you don't have access to academia. :-(
Thankfully research is not guided by notions like "what makes money in the current economy"; otherwise we'd be stuck with faster calculators instead of, well, all of modern computer science.
> Thankfully research is not guided by notions like "what makes money in the current economy"
Unfortunately I don't believe this is true for AI research... I think you'll find a strong correlation with each year's most cited papers and what's currently popular in industry. There's always exceptions, but we moved to a world where we're highly benchmark oriented, and that means we're highly reliant on having large compute infrastructures, and that means that the access and/or funding comes from industry. Who is obviously going to pressure research directions towards certain things.
I did not say anything about making money in the current economy. Research requires funding and only way to have academic freedom is to have a steady source of recurrent income or funding. One way to get that is to please those who have money but give up part of your control on what you study. Other way is to monetize your knowledge and acquire a ton of money. You seem to misunderstand what I said or have very little clue on how the academia works.
Although point #1 is true, I find it to be a slightly superficial advice. It's like saying "if you want to be happy, find a job that fulfills you". Sure, everyone wants to be able to focus on good projects before papers, so that the papers come naturally. If you can do that, congrats! You won the game; be proud and enjoy it. However, the truth is that the way to get there is dark and lonely and full of terrors, and not everyone can do it. All academics (especially junior ones) are constantly in the middle of an ambiguous, contradictory discourse: you must produce a lot, but you also must produce high-quality output. Your environment, literally everyone around you, wants to have your cake and eat it too. As you get more experienced you learn to navigate this and keep the good parts while ignoring the bad ones, but for young researchers this can be extremely taxing and stressful. In order to "focus on projects and not papers" you have to literally swim against the current, usually in a very very strong current. Not everyone has the presence of mind and attitude to do it, and to be honest, it's not fair to expect that either.
So, here are some points and comments I offer that go in a slightly different direction (although, like I said, if you managed to get there, congrats!):
* You can write a good paper without it being a good project. One thing does not exclude the other, and the fact that there are many bad papers out there does not mean that papers themselves are bad. You can plan your work around a paper, do a good research job, and write a good scientific report without having to have an overarching research project that spills over that. Sure, it is great when it happens (and it will happen the more experienced and senior you get), but it's not necessarily true.
* Not thinking about the paper you'll write out of your work might deter you from operationalizing your research correctly. Not every project can be translated into a good research paper, with objective/concrete measurements that translate to a good scientific report. You might end up with a good Github repo (with lots of stars and forks) and if that's your goal, then great! But if your goal is to publish, you need to think early on: "what can I do that will be translated into a good scientific paper later?" This will guide your methods towards the right direction and make sure you do not pull your hair later (at least not as many) when you get rejected a million times and end up putting your paper in a venue you're not proud of.
* Publishing papers generates motivation. When a young research goes too long without seeing the results of their work, they lose motivation. It's very common for students to have this philosophical stance that they want to work on the next big project that will change the world, and that they need time and comfort and peace to do that, so please don't bother me with this "paper" talk. Fast forward three years later they have nothing published, are depressed, and spend their time playing video games and procrastinating. The fact is that people see other people moving forward, and if they don't, no amount of willpower to "save the world" with a big project will keep them going. Publishing papers gives motivation; you feel that your work was worth it, you go to conferences and talk to people, you hear feedback from the community. It's extremely important, and there's no world where a PhD student without papers is healthier and happier than one with papers.
* Finishing a paper and starting the next one is a healthy work discipline. Some people just want to write a good paper and move on. Not everyone feels so passionate about their work that they want to spend their personal time with it, and push it over all boundaries. You don't have to turn your work into your entire life. Doing a good job and then moving on is a very healthy practice.
> how do I do research that makes a difference in the current, rather crowded AI space?
I hope they are first asking, to which bank accounts is the research actually making a difference? It's a great fraud to present research problems to stimulate the intellect of the naive youth when they have no capability to assess its social impact.
"Invest in projects, not papers." is the typical thing you hear from late-stage researchers that forgot what it's like to be a PhD student or early post-doc. We have to publish as much as we can, or our supervisors and committees won't allow us to progress in our careers.
I can't spend a year on topics that seem interesting but that might not yield papers if they don't work. From the bureaucratic point of view, which is almost all that matter for junior researchers, that would be simply time in the bin.
I would love to spend years on something I care about without caring how many papers it will generate, but if I do that, I won't have a career.
I actually did invest in projects rather than papers and I definitely feel I paid the price. It's the main reason I opted for a postdoc rather than going straight on the academic market: the quality of my research was high, but the number of publications I have is too low to be competitive. At least that's what my PhD advisor said and I honestly agree, especially after speaking with people who just landed tenure-track jobs and more senior professors.
What I have to say may come across as harsh but I only intend it to be helpful.
If you invested in projects, and the investments did not pay off, I see three likely explanations.
First, maybe your investment function is miscalibrated.
Second, maybe your investment function is well calibrated but your time horizon is too long.
Third, maybe you are unlucky.
I have no insights into what happened in your case, I don't even know what field your are in.
But, case 1 suggests you may be unsuccessful in academia.
Case 2 suggests possible success after adjustment to more immediate reward.
Case three suggests possible success after a difficult recovery process and further investments.
None are ideal paths but cases 2 and 3 suggest possible recovery strategies if you are deeply committed to academia. The optimality of such an approach is highly subjective.
You likely should not feel like academia is the only available path, if you do that is a red flag that something is amiss.
What about 4: Investing in projects over papers is a fundamentally bad early career strategy in modern academia?
> I won't have a career
in academia.
This whole system seems like something out of a bizarre comedic parody of human society, except it's real.
I don't think these big AI labs give you much credit for being a PhD dropout
They do... if you have NeurIPS/ACL/EMNLP publications! :)
Even then, there are too many PhDs for few spots in academia and top-tier labs, so there's no way a PhD dropout could compete unless they have some other kind of experience.
Also, it's not that hard to publish in high-end NLP conferences, so it doesn't say much.
I watched a startup give an NLP AI engineer with only 1 workshop paper at ACL get a 150K remote offer less than a month ago. This person is in the middle of their BS program right now.
It is most certainly not easy to publish top AI research (unless you are doing unethical things). I repeat, if you have a NeurIPS main conference publication, you don't need to have a degree for a top AI lab to at least consider you.
If your experiences are different, my guess is that you're not an American.
No, not an American. Did you mean that Americans have lower standards than Europeans in this respect? I'm confused about your last sentence.
Anyway, my note was specifically about publishing NLP research at ACL conferences (ACL, NAACL, EMNLP, etc.), where publishing mediocre work hasn't been difficult. Just go through the proceedings, and you'll see that's the case. I don't know about NeurIPS.
> No, not an American. Did you mean that Americans have lower standards than Europeans in this respect? I'm confused about your last sentence.
Perhaps the US has more flexible hiring standards? Given all the school dropout success stories here. As apposed to lower standards.
This has good advice about academic research in general not just AI Research. It's huge to "Select timely problems with large headroom and "fanout"" - you need a research program to have a successful research career. At least most people do. It's also huge to "Put your work out there and own popularizing your ideas" - the second part is hard for a lot of academics to do. Putting your work out there is huge to get feedback, which may help you pivot or try different directions, as well as fostering collaboration. Collaboration is a large part of being a successful researcher.
> Collaboration is a large part of being a successful researcher.
I would even consider it as the largest part. It brings ideas, funding, spreading of your ideas, access to institutional research grants/projects, industry contacts, and more.
Agreed! It also brings friendships, new ways of thinking and framing, inspiration, and the opportunity to give and receive support - to discover how much you can matter, and the good you can do.
Completely second that and the parent comment!
Not to mention it’s a good way to have an excuse for a glass of champagne or wine when you finally meet remote co-authors.
The lifeblood of a successful research career.
I think many great ideas never gain traction simply because they weren’t communicated effectively
> you need a research program
What is a "research program"?
The idea is that rather than doing a bunch of completely independent research projects, all of your projects are designed in service of answering some larger research question.
the "concepts of a plan"
At least some of it comes from "hype" too. The author of Dspy (the writer) (https://github.com/stanfordnlp/dspy) should know this, given that Dspy is nothing more than fancy prompts optimizing prompts to be fancier according to prompt chains described in papers (i.e. Chain of thought, Tree of thought, etc). Textgrad (https://github.com/zou-group/textgrad) is an even worse example of this, as it makes people think that it's not just a prompt optimizing another prompt
Dspy has 17k stars, meanwhile PyReft (https://github.com/stanfordnlp/pyreft) isn't even at 1200 yet and it has Christopher Manning (head of AI at stanford) working on it (see their paper: https://arxiv.org/abs/2404.03592). Sometimes what the world deems "impactful" in the short-medium term is wrong. Think long term. PyReft is likely the beginning of an explosion in demand for ultra parameter efficient techniques, while Dspy will likely fade into obscurity over time.
I also know that the folks writing better samplers/optimizers for LLMs get almost no love/credit relative to the outsized impact they have on the field. A new sampler potentially improves EVERY LLM EVER! Folks like Clara Meister or the authors of the min_p paper preprint have had far larger impacts on the field than their citation counts might suggest, based on the fact that typiciality or min_p sampling is now considered generally superior to top_p/top_k (OpenAI, Anthropic, Gemini, et al still use top_p/top_k) and min_p/typicality are implemented by every open source LLM inference engine (i.e. huggingface, vllm, sglang, etc)
I think that's a little harsh. Imo the reason for difference in popularity/github-stars is just different user bases- an order of magnitude more people use LLM APIs (and can leverage Dspy) vs those who finetune an LLM.
Agree about the abstractions btw. I found Dspy very convoluted for what it does, couldn't make sense of Textgrad at all.
I like the article directionally but fear the examples are too short-sighted for most AI researchers.
Picking something useful in 1-2 years is a reason to go to industry, not research, and leads to mostly incremental units that if you don't do, someone else will. Yes, hot topics are good because they signal a time of fertile innovation. But not if your vision is so shallow that you will have half of IBM, Google, and YC competing with you before you start or by the time of your first publication (6-12mo). If you are a top student, well-educated already, with top resources and your own mentees, and your advisor is an industry leader who already knows where your work will go, maybe go to the thickest 1-2 year out AI VC fest, but that's not most PhD students.
A 'practical' area would be obvious to everyone in 5 years, but winnowing out the crowd, there should not be much point to it today nor 1-2 years without something fundamentally changing. It should be tangible enough to be relevant and enticing, but too expensive for whatever reasons. More fundamental research would be even more years out. This gives you a year or two to dig into the problem, and another year or two to build out fundamental solutions, and then a couple years of cranking. From there, rinse-and-repeat via your own career or those of your future students.
Some of my favorite work took 1-2 years of research to establish the problem, not just the solutions. Two of the projects here were weird at first as problems on a longer time scale, but ended up as part of $20M grant and software many folks here use & love, and another, a 10 year test of time award. (And another, arguably a $100M+ division at Nvidia). In contrast, most of my topic-of-the-year stuff didn't matter and was interchangeable with work by others.
Edit: The speech by Hamming on "You and your research" hits on similar themes and speaks more to my experiences here: https://fs.blog/great-talks/richard-hamming-your-research/
I find that in AI / ML / computer vision, the thinking horizon has shrunk a lot. Reviewers prize straightforward incremental work that nevertheless beats the SOTA. New problem formulations are too hard to decipher and understand deeply and nobody has time nowadays. Everything is super sped up. There's a fire hose of papers coming out that all do slightly different things, or the same concurrently.
In this area even the best, most respected professors don't have long term laid out research plans. Grant plans must exist but they never quite pan out that way. Researchers in this environment are very reactive and it's all about being fast. Diffusion is popular? Then let's do diffusion for [my specialty], or transformers for [X]. Or combine LLM with [my last topic]. People are constantly pivoting and jumping from one opportunity to the next. In my experience gone are the days where you could carve out a niche and work in peace on it for 4-5 years and have a consistent overarching story in your PhD.
IMO that kind of thinking flips correlation/causation, and feels like some sort salience bias towards voluminous but crappy work. Yes, you can do it, just good bye impact. Worse, it's a self-fulfilling prophecy: by moving your work to be more incremental, you're even more at risk by the incremental advances of others.
The field exploded, so yes, there are way more good and even more bad papers, so some researchers interpret that as the need to publish a lot. Top labs attract a lot of students and funding, so the PI spends little time per student, and instead farms out to post docs who are there for only a few years, and in turn they do most of the phd student mentoring. And many folks want to bean count neurips papers. The result is yes, if you're unintentional about impact, and get distracted by the volume, you too can optimize for that, do lets of incremental work, ultimately you've learned some skills and wrote papers that won't matter. Google/FB/OpenAI/Spotify/LinkedIn/etc will still hire you at the end, that's fine, they hire a ton of phds.
Objectively, there are some real jumps happening, but zoomed out, very few of the papers coming out in a given year change all that much. E.g., we're using LLMs for some hard societal problems (misinfo, cyber, ...), but nothing fundamental changed in our startup's work since gpt-4 came out, and all versions before gpt-4 weren't significant enough to motivate using generative LLM methods. There are a few methods we're tracking (e.g., graph RAG, certain kinds of agents), but most work is embarrassingly shallow or bad, so the incremental papers have done almost nothing to influence us, and we learn more from other industry folks doing more ambitious or real work here. The previous disruption on our AI side was around GNNs, but that was over many years and quite rocky. So if we were doing all this as a serious PhD project (invention) instead of as a startup (innovation), I'd expect similarly low levels of disruption from the perceived fast pace of others.
True, but at the end of the day, a mediocre PhD that gets you a nice job is, all things considered, not a bad outcome from the point of view of the student. Yes, obviously becoming MrBeast is better than being the (n+1)th gaming streamer. But I think some might not realize how many PhDs are churned out constantly. PhDs are also given out by mediocre universities, advised by mediocre profs. Not everyone can be the next AI superstar.
Beware the extreme selection/survivorship bias of listening to advice from the top most successful researchers and household names / celebrities. Most PhDs make a small dent in a small niche specialty, get their degree and go on with their lives, and their work is superseded 6 months to 2 years later.
If no one did the research that anyone could do, it would never get done.
As I wrote, different folks will already be chasing more immediate projects as they're too attractive: Think masters students, industry labs, startups, etc.
Rules are meant to be broken. For example, a professor will often put a masters student on a cute me-too one-off project in a hot area as the short-termism helps the student have a smoother time, and the professor is trading expected low impact in the the project for a cheap shot to see if there is a more interesting question behind the immediate one.
Much of the PhD is about learning to spot & navigate problems, not solutions. Young PhD students can totally pick the same me-too problems of masters students. The advisors would likely just think the student is using the same approach to seeing where the PhD worthy problem is. If still nothing, should be ready to drop the topic, or if they truly love me-too work more than the PhD process, leave for a better environment for that, like a company / industry lab / startup throwing more resources at the problem.
Also, not every phd needs to do the original topic of attempting impact. You can totally use the PhD years to learn how to do me-too & unimpactful work or even the most weird & niche problems: as long as at least one advisor thinks they can stay funded while supporting you, most programs will let you through.
You're right, but the individual has to ask if entering a crowded niche is worth it or not. Does your main strength lie in absurd workload to grind out the paper a few months/weeks earlier to Arxiv than the next group will?
The risk reward calculus still often benefits this approach. The risk is that you get scooped. If you do something nonobvious, the risk is that it doesn't pan out or the reviewers don't get the point. The latter is harder to solve by grinding. So many phds will chose the former, as at least putting in ridiculous hours feels more actionable. Also, doing the straightforward thing and getting scooped is less ego-hurting than failing at your own special pet idea.
I think many who aren't in grad school don't understand how extremely important it is to have publications. At the beginning you must get publications to do internships and to build contacts at conferences. Then later it becomes imperative in order to graduate and get your next position. Nowadays you must even have multiple top conference papers just to start a PhD. Of course from the point of view of the advisor with 15 PhD students, the ideal student would do high risk high reward work and if 3 projects become super successful, 5 become quite successful and 7 yield nothing, that's still great for the advisor. But the 7 who lost several years would rather not go through this.
In most markets, it pays to be contrarian.
... and when everybody decides to be a contrarian, the contrarian position is to do the popular thing. But the catch is that this never happens on practice.
Unfortunately while this advice sounds useful, it isn’t. It might be useful if you measure impact is the fake metrics like citation count but if you want to measure with actual tangible impact on the real world, you have a rude awakening before you. To me there are 2 ways a paper can create impact:
1. The paper represents such an impressive leap in performance over existing methods in AI, that it is obviously impactful. Unfortunately, this way of generating impact is dominated by industry. No one can expect Academia to train O1, SAM, GPT5 etc. AI rewards scale, scale requires money, resources and manpower and Academia has none. In the early days of AI, there were rare moments when this was possible, AlexNet, Adam, Transformers, PPO etc. Is it still possible? I do not know, I have not seen anything in the last 3 years and I’m not optimistic many such opportunities are left. Even validating your idea tends to require the scale of industry.
2. The paper affects the thought process of other AI researchers and thus you are indirectly impactful if any of them cause big leaps in AI performance. Unfortunately here is where Academia has shot itself in the foot by generating so many damn papers every year (>10,000). There are just so many, that the effect of any 1 paper is meaningless. In fact the only way to be impactful now is to be in a social circle of great researchers, so that you know your social circle will read your paper and later if any of them make big performance improvements, you can believe that you played a small role in it. I have spoken to a lot of ML researchers, and they told me they choose papers to read just based on people and research groups they know. Even being a NeurIPS spotlight paper, means less than 10% of researchers will read your paper, maybe it will go to 50% if it’s a NEURIPS best paper but even that I doubt. How many researchers remember last year’s NEURIPS best paper?
The only solution to problem 2, is radical. The ML community needs to come together and limit the number of papers it wide releases. Let us say it came out and said that yearly only 20 curated papers will be widely published. Then you can bet most of the ML community will read all 20 of those papers and engage with it deeply as they will be capable of spending more than a day at least thinking about the paper. Of course you can still publish on arxiv, share with friends etc but unless such a dramatic cutdown is made I don’t see how you can be an actually impactful AI researcher in Academia when option 1 is too expensive and option 2 is made impossible.
> Academia has shot itself in the foot by generating so many damn papers every year (>10,000).
Each PhD student needs 3-5 papers to graduate. Nowadays you even need 1-2 to simply get into a PhD program. The huge number of papers compared to, say, 10 or 15 years ago simply reflects how many more PhD students there are in this area.
It definitely isn't sustainable but academia is a distributed system. There's no way to coordinate publishing less. No individual agent will rationally choose that. Especially that it's not just competition within AI but other CS fields too for huge grants and the grant agencies defer to citation and publication metrics. CVPR is among the top scientific publication venues by various metrics and this state of affairs also benefits the field.
There are reasons for why things are this way but the way it's going is clearly unsustainable. It's a red queen race, similar to grade inflation, credential inflation, and various forms of cost disease.
With one outlier, by the one academic AI lab that continues to produce impactful research every year, Christopher Ré's at Stanford (although the raised VC money now so who knows), FlashAttention.
If you were a really good academic, you could come up with a theory that predicts the performance of a model you haven't actually tested. Physics is full of this - theories that predict something novel followed by other people doing expensive experiments to test them. The guy who published the theory gets the credit. That's probably what we want from academic AI researchers - theories that we can eventually use to design models, rather than just haphazardly building something and wondering how good it will be.
That theory sounds great. There's no such thing at the moment and maybe never. Lots of smart people have tried.
Just because that's what people want, doesn't mean we can produce it. I often talk to funding agencies about things like this. "We don't want to fund boring research, only what will give us the ultimate theory of how everything works". That's not how science or progress work.
> That theory sounds great. There's no such thing at the moment and maybe never. Lots of smart people have tried.
In my opinion the problem rather is that considering the current AI gold rush, people are rather eager to throw newly implemented models around instead of thinking really deeply how an insanely better model could look like.
"Invest in projects, not papers."
This is excellent advice, and in my experience does not represent the intuition that many young (and not so young) researchers begin with.
Papers come from projects and, if you care, good projects can yield many good papers!
The problem is my supervisor only cares about paper count.
Article's point seems to be that in the long term: paper count, citations, impact, motivation and fulfillment will all come from focusing on a project.
I tend to agree but there are way too many paper mills out there and Ive been stuck in one.
The gamification of google scholar is real
Idk, nothing wrong with going for a low-hanging fruit and doing a one-off sometimes. So many academics fail to get stuff over the finish line. Not the right advice for everybody.
Yeah, but it's not saying "don't do papers", it's saying your long-term investment should be projects.
Interesting. This is a common problem in academia?
Depends on the person. Not unheard of to see people with 10+ working papers that realistically will never see the light of day. Each of those are months of work.
I can see that I guess. Easier to start a new one than do the annoying parts to finish one.
nothing wrong except that it might be a distraction, which sometimes is good and sometimes would be better avoided.
Focusing on quantity over quality is nearly always a bad idea
This article sort of lists the ideals, but is factually contradicted by the immense amounts of non-impactful and trivial AI papers, who have around 20 authors each.
There does seem to be a strong incentive to publish whatever and distribute the credit among dozens of people.
For the rare actually impactful research the advice is a bit trivial, you might as well quote Feynman:
Pet peeve: you're quoting not-Feynman.
For context, Gell-Mann just said this was how Feynman solved problems, Feynman didn't say it.
(Coincidentally it's the same case as the term "Gell-Mann Amnesia".9
https://pca.st/rr17x2vb
A lot of the advice is good but this is sad:
> First, the problem must be timely. You can define this in many ways, but one strategy that works well in AI is to seek a problem space that will be 'hot' in 2-3 years but hasn't nearly become mainstream yet.
I think about research as, what can I bring that's unique? What can I work on that won't be popular or won't exist unless I do it?
If it's clearly going to become popular than other people will do it. So why do I need to? I'm useless.
Yes. You'll increase your citation count with that plan. But if you're going to do what other people are doing go to industry and make money. It seems crazy to me to give up half a million dollars a year to do something obvious and boring a few months before someone else would do it.
Artificial intelligence is becoming part of everyday life, and Apple’s advancements are a great example. Submagic https://ai-depot.net/tools/submagic/ is one tool that stands out for automatically transcribing speech to text, making content more accessible. It’s especially helpful for video creators looking to reach wider audiences across different languages. The tool handles transcription while you can focus on the creative aspects. It’s a time-saver that can enhance content quality with little extra effort.
Academic research in the USA today is entirely corporatized. Try publishing a paper on fine-tuning LLMs with the goal of creating agents capable of replacing the executive suite and the board of directors for major corporations and see what happens to your career. It's far worse today than what Eisenhower described c. 1960 in his final address to the nation, because so much of the research money is now doled out by corporate entities:
> "Today, the solitary inventor, tinkering in his shop, has been overshadowed by task forces of scientists in laboratories and testing fields. In the same fashion, the free university, historically the fountainhead of free ideas and scientific discovery, has experienced a revolution in the conduct of research. Partly because of the huge costs involved, a government contract becomes virtually a substitute for intellectual curiosity. For every old blackboard there are now hundreds of new electronic computers. The prospect of domination of the nation's scholars by Federal employment, project allocations, and the power of money is ever present — and is gravely to be regarded."
Unless you are independently (very) wealthy, you'll have to align your research with the goals of the funding entity, be that the corporate sector or a government entirely controlled by the corporate sector. You may find something useful and interesting to do within these constraints - but academic freedom is a myth under this system.
> Try publishing a paper on fine-tuning LLMs with the goal of creating agents capable of replacing the executive suite and the board of directors for major corporations and see what happens to your career.
I expect if you manage to get some interesting results, your career will be great.
The trouble is that there aren't any "official" benchmarks for evaluating performance for such a model. That's the main problem and makes it very hard to show if your models are any good.
A lot to like here, a couple of things to comment on:
> "If you tell people that their systems could be 1.5x faster or 5% more effective, that's rarely going to beat inertia. In my view, you need to find problems where there's non-zero hope that you'll make things, say, 20x faster or 30% more effective, at least after years of work."
This works for up-and-coming fields, but once something is stable and works at large scale, it's all about the small improvements. Making petrol engines 1% more fuel-efficient would be massive. Increasing the conversion rate of online ads by 1% could make you very, very rich indeed. Good advice for AI probably; bad advice in other fields.
> "Invest in projects, not papers"
The best way I think you can go about this is allocate some fraction alpha of your time to projects, and (1-alpha) to things that produce short-term papers. Alpha should never be zero if you want a career, but it will start out small as you begin your PhD and gradually grow, if you can make it in academia. At some point you'll reach a compounding return where the projects themselves are spawning papers - one way to do this is to get to the point where you can hire your own PhD students, but there are several others.
As long as your 2-years-into-a-PhD review as some unis have them is about how many papers you've published (somehow weighted by journal/conference rank) and how many others are in the pipeline, you need to focus on papers until the point when your institution will let you do something more useful. Think of it as paper writing bootcamp so that once you do get more time for projects, you'll have practiced how to write up your results.
> "Make your release usable, useful ..."
This is excellent advice, also for anything else related to code.
> Increasing the conversion rate of online ads by 1% could make you very, very rich indeed.
I claim there is actually a rather easy way to do this (but it won't make you rich!).
The basic idea is rather simple: for each potential (ad/product, user) pair, ideally store two kinds of information:
- the estimated probability that the user will click on the ad and/or make a buying decision
- the confidence that you have in the correctness of this estimate
Then only show ads where, based on this data, the conversion rate will be high with a high probability.
Result: by only showing ads to those few people who very likely want to buy your product, you create an insanely high conversion rate, but you loose money because the ad will be shown to rather few people (possibly these are even people who don't actually have to get "convinced" to buy your product).
In other words: we just created an artificial example of Goodhart's law ("When a measure becomes a target, it ceases to be a good measure") in action. What this lesson tells us is
- the "conversion rate" of an online ad is just a proxy for some entirely different business goal that you have
- the conversion rate can rather easily be manipulated
> Making petrol engines 1% more fuel-efficient would be massive. Increasing the conversion rate of online ads by 1% could make you very, very rich indeed. Good advice for AI probably; bad advice in other fields.
It's good advice in every field. Most improvements have a cost. If you make engines 1% more efficient but 5% more expensive no one will care.
Heck electric engines are wildly more efficient (80-95%) compared to combustion engines (20-35%), not to mention far simpler, and we can hardly get people to switch. Even immense improvements can be hard to roll out.
Don't work on marginal gains. Any minor problem or inconvenience will wipe them out and all your time and effort will be worth nothing. Let people who are paid by companies do that.
If you're going to do a PhD focus on something that could be big. Whenever students suggest a project I always start by asking what the upside is: what if we massively succeed?
Is this article roughly the application of the startup mindset to research? With the major downsides that you're giving away your insights for free, and potentially actively help other companies profit from your work. You don't go as far as to productize the research, but just make it really easy for others to do so by building "artifacts" for it.
I'm not a researcher, but have thought about doing a PhD in the past.
It's probably a lot more nuanced that this. Show progress but don't make it easily accessible. *Hide* something important for yourself. Kind of like modern day "open source".
This used to be very common but nowadays it's more and more expected to release the code with the paper. This makes it harder to obfuscate big parts of the method. Arguably, this also has lowered the bar for doing incremental research. Just get the code of the sota from github, implement a new feature, test it, if it's 1% better then publish it.
Previously there was a natural gatekeep ing effect that you had to be able to understand and translate to code all the math and equations in the paper and be familiar with the dark knowledge details that everyone in the field knows but isn't spelled out explicitly in the papers. Boring bits like how to exactly preprocess, normalize, clean the data, how to precisely do the evals etc.
> Is this article roughly the application of the startup mindset to research?
Somewhat but a few crucial points are missing for it to be that. For example,
- hire a lot more people (PhD candidates, postdocs) than you have funding to pay till they finish. Keep growing and getting more grants until you get a Nobel prize (or equivalent) or go broke
- fake your data and publish in the top journals/conferences
- don't do any research yourself, just get others to do everything and "coordinate" them. Make sure you get all the credit for the publication. The way to get others to do things is to "collaborate" with them or maybe (promise to) pay them.
Once you got that ball rolling your name becomes a "brand" and collaborators will beg you to put your name on their papers so it gets more exposure.
This advice is much more general than academic research. It applies to startups as well.
I would say follow the money, create a product or service that generates recurrent income and channel that income to study the fundamental problems that would interest you. Owning a product or platform opens up for a lot of opportunities.
This would require a potentially successful researcher to also be a successful entrepreneur. Seems like it does not solve the problem for most people.
Also, even with a steady income, not having access to academia can get in the way of successfully advancing research.
Still, I do agree that it's a great option to buy yourself some freedom, if you are capable of doing so.
I don’t think you would loose access to academia, actually it is the opposite. Academia would invite you with open arms once you have funds and/or the data :)
> Academia would invite you with open arms once you have funds and/or the data :)
This way of getting access to academia is based on the premise that you already did quite some research - which is much harder if you don't have access to academia. :-(
If we are referring to PhD positions, that’s the case anyway. Many masters/PhD applicants have multiple papers these days.
Thankfully research is not guided by notions like "what makes money in the current economy"; otherwise we'd be stuck with faster calculators instead of, well, all of modern computer science.
I did not say anything about making money in the current economy. Research requires funding and only way to have academic freedom is to have a steady source of recurrent income or funding. One way to get that is to please those who have money but give up part of your control on what you study. Other way is to monetize your knowledge and acquire a ton of money. You seem to misunderstand what I said or have very little clue on how the academia works.
>Thankfully research is not guided by notions like "what makes money in the current economy"
That's right, it's guided by "what gets grant money in the current paradigm"
Most research I know about in academia is absolutely about what makes money.
Care to give some examples? It is largely untrue in CS academia, and definitely untrue in every other non-engineering field.
Although point #1 is true, I find it to be a slightly superficial advice. It's like saying "if you want to be happy, find a job that fulfills you". Sure, everyone wants to be able to focus on good projects before papers, so that the papers come naturally. If you can do that, congrats! You won the game; be proud and enjoy it. However, the truth is that the way to get there is dark and lonely and full of terrors, and not everyone can do it. All academics (especially junior ones) are constantly in the middle of an ambiguous, contradictory discourse: you must produce a lot, but you also must produce high-quality output. Your environment, literally everyone around you, wants to have your cake and eat it too. As you get more experienced you learn to navigate this and keep the good parts while ignoring the bad ones, but for young researchers this can be extremely taxing and stressful. In order to "focus on projects and not papers" you have to literally swim against the current, usually in a very very strong current. Not everyone has the presence of mind and attitude to do it, and to be honest, it's not fair to expect that either.
So, here are some points and comments I offer that go in a slightly different direction (although, like I said, if you managed to get there, congrats!):
* You can write a good paper without it being a good project. One thing does not exclude the other, and the fact that there are many bad papers out there does not mean that papers themselves are bad. You can plan your work around a paper, do a good research job, and write a good scientific report without having to have an overarching research project that spills over that. Sure, it is great when it happens (and it will happen the more experienced and senior you get), but it's not necessarily true.
* Not thinking about the paper you'll write out of your work might deter you from operationalizing your research correctly. Not every project can be translated into a good research paper, with objective/concrete measurements that translate to a good scientific report. You might end up with a good Github repo (with lots of stars and forks) and if that's your goal, then great! But if your goal is to publish, you need to think early on: "what can I do that will be translated into a good scientific paper later?" This will guide your methods towards the right direction and make sure you do not pull your hair later (at least not as many) when you get rejected a million times and end up putting your paper in a venue you're not proud of.
* Publishing papers generates motivation. When a young research goes too long without seeing the results of their work, they lose motivation. It's very common for students to have this philosophical stance that they want to work on the next big project that will change the world, and that they need time and comfort and peace to do that, so please don't bother me with this "paper" talk. Fast forward three years later they have nothing published, are depressed, and spend their time playing video games and procrastinating. The fact is that people see other people moving forward, and if they don't, no amount of willpower to "save the world" with a big project will keep them going. Publishing papers gives motivation; you feel that your work was worth it, you go to conferences and talk to people, you hear feedback from the community. It's extremely important, and there's no world where a PhD student without papers is healthier and happier than one with papers.
* Finishing a paper and starting the next one is a healthy work discipline. Some people just want to write a good paper and move on. Not everyone feels so passionate about their work that they want to spend their personal time with it, and push it over all boundaries. You don't have to turn your work into your entire life. Doing a good job and then moving on is a very healthy practice.
Is there any open-source project recommended that be kind to deep learning newbies that can learn or contribute to?
i don't fully agree with the headroom, many papers were published achieving 1% improvement on ImageNet...
Agreed. Any improvement in a well established and competitive metric which a community is tracking is going to get attention and respect.
[dead]
[dead]
> how do I do research that makes a difference in the current, rather crowded AI space?
I hope they are first asking, to which bank accounts is the research actually making a difference? It's a great fraud to present research problems to stimulate the intellect of the naive youth when they have no capability to assess its social impact.