The Cliffs of Insanity: Dramatic Shifts in Technologies on Stack Overflow
Life as a developer (or data scientist, in my case) involves being comfortable with changing technologies. I don’t use the same programming languages that I did at the beginning of my career, and I fully expect to be using different technologies several years from now. Here on the Stack Overflow blog, we’ve talked recently about technologies that are shrinking in popularity and those that are growing, but sometimes a programming language, framework, or technology comes out of nowhere and bursts on to the scene, or falls off a cliff. Sometimes there are dramatic shifts in a technology’s impact on the software industry. In this blog post, let’s explore the most dramatic rises and falls in Stack Overflow questions by tag since the site’s beginning.
Most dramatic increases
To find the biggest growth, let’s look at year over year change in questions asked for tags in Stack Overflow. What tags have the highest percent change, in any year?
This analysis includes Stack Overflow tags with at least 10,000 questions and excludes tags that end in numerals like ios8.
Two tags stand out in this analysis, both with tremendous growth, and they have something in common. Swift is Apple’s language for developing iOS apps that is a successor to Objective-C, and the angular tag on Stack Overflow is used for newer releases than the angularjs tag. Both of these technologies grew incredibly fast to have a big impact because they were natural next steps for existing developer communities. (Note that AngularJS also grew incredibly fast, though, and has the 10th highest rate of year-over-year growth ever on Stack Overflow.)
The other technologies here exhibit comparatively modest year-over-year growth, but are all interesting examples of what kinds of things can happen in our industry.
Android Studio is an IDE targeted to mobile developers; it was first released in 2013 and grew immediately in popularity. The angularjs-directive tag is obviously related to the massive growth of Angular. The iPad was announced by Apple in 2010, and the machine learning library TensorFlow was released as open source by Google in 2015. We see from this plot what remarkable success looks like for a new technology.
Most dramatic decreases
What about the other side of the coin? What tags have the largest percent decreases, in any given year?
Like the first plot, this includes Stack Overflow tags with at least 10,000 questions and excludes tags that end in numerals like ios4 and xcode6.
Backbone.js is a JavaScript framework, with the brutal lifecycle typical for that ecosystem, and the game engine Cocos2d for iPhone has fallen out of favor. Silverlight has been deprecated by Microsoft, and Flex is a framework for Flash, also on its way out. Both were featured in our post on shrinking technologies from August. The Object-Relational Mapping tag has a higher level of post-peak activity than the other tags here, but is dramatically lower than about a decade ago.
Parse is a pretty interesting example here. Parse is a mobile backend developer platform that was acquired by Facebook in 2013; Facebook announced it was shutting down the platform in (you can probably guess this) January 2016. Parse was open sourced in the meantime. I like Parse as the flipside of the examples like Swift where a new technology is announced; Parse was a growing, apparently popular platform and its shuttering was announced.
Sustained growth
Technologies can exhibit dramatic shifts like we’ve discussed, but some technologies grow at a more steady rate. What Stack Overflow tags have exhibited the most sustained growth since 2010, with questions in at least 5 years? Let’s look at the median annual growth for tags with at least 10,000 questions and find technologies that have grown at a high level over many years, not only in a burst.
Here we see the first Angular tag, along with other JavaScript ecosystem members like TypeScript and Meteor. Several of these technologies are connected to the growth of data science and machine learning, including Pandas and the dataframe tag. Others occupy unique positions in the software industry, such as the ubiquitous search engine Elasticsearch and the game engine Unity. These technologies are diverse, but they all have grown at strong and steady rates over the past 5 to 7 years. Check out the Stack Overflow Trends tool to see what their growth patterns look like.
Here at Stack Overflow, we understand how technologies in the software industry rise and fall, and we have data to see these changes first hand. We use our data and expertise to help our clients make better decisions when engaging with developers. Check out how we can help you!
51 Comments
I’m not sure how dramatic it is that Swift usage skyrocketed in 2014, prior to which it did not exist.
Well, that’s… the point. I’m not trying to be facetious here. I looked at lots of technologies and their growth rates. Which grew the fastest? Most technologies do NOT grow as fast as these. You can check out some, if you like, like Go: https://insights.stackoverflow.com/trends?tags=go
(Not the persion you replied to) I get your point, but I think its a somewhat different thing when its basically a corporate sponsored replacement. I can’t help but wonder if the objective-c -> swift transition is more similar to the ios6->7->8 changes (which were excluded) than it is to say, the rise of Go. Might be an interesting analysis on its own (could also compare say, how fast android versions replaced their predecessors & how the angular/angularjs crossover)
That is a really fair point, and I did try to point that out in the post. Both Swift and Angular were natural next steps for existing developer communities, not something entirely new.
Good research. Though number of questions asked is a proxy metric for platform growth and at times represents level of confusion that developers have. The fact that we see JS stuff on top of the so called growth list is remarkable on it’s own.
I’m pretty sure Angular skyrocketed because developers coming from AngularJS to Angular 2 where confused as hell, also documentation is lacking and full of “foo bar” examples (pretty useless).
Trends isn’t working right: https://meta.stackoverflow.com/questions/359295/stack-overflow-trends-doesnt-work
What are people using instead of ORM? Raw SQL?
That actually is what’s happening with a project I’m contributing to, but they’re looking to move away from that.
Other patterns like Repository pattern
The Repository Pattern is just an abstraction layer over a Data Mapper. Unless you plan on writing your entire program in SQL, you still need something that converts database records into objects for your program’s consumption. See Fowler’s Data Mapper and Repository articles.
It was an attempt at a summary. I thought you were asking a question about what other people might be using in place of an ORM, as a discussion. Yes, you still need a data mapper but we’ve written ours ourselves which works quite well, provides a pattern each time you need to add a domain object. We have a Service[Class] as an interface to each repository. It is extremely easy to mock up the data coming from the repository. All our SQL is in stored procs — no dynamic sql. There’s much more to it, but it works well as it has matured over time.
While the rest of the tags are tracking implementations, orm is an idea. People now have better tags; regarding just Java we’ll find “hibernate”, “eclipese-link”, “jpa”… adding the “orm” tag really doesn’t get you anything. You want to know how to get it working on a specific implementation. I highly doubt there is a decline in the technology but rather there is a multitude of better tags to pick from. The rest of the tags are more less specific to an implementation (version numbers aside), so their tracking should be rather accurate.
Thank you for not putting a big graphic at the beginning of your post.
I’m glad to see flex and silverlight declining so steeply, though I wonder what kind of growth replacement tags are seeing.
Are you sure that first graph is right? The graph says “angularjs-directive” not “angularjs”
https://insights.stackoverflow.com/trends?tags=angular%2Cangularjs%2Cangularjs-directive
https://uploads.disquscdn.com/images/53adf844cad4b69f39df4b9e6207d632ff4b9c4562bebef5eb1f047827c446cf.png
Yep, I am sure! 🙂 This post analyzed the year-over-year growth here on Stack Overflow, not the absolute increase. You can think of that as asking the question, “Which technologies grew the most, relative to how big they were the year before?” Notice in my post that I distinguish between the angular and angularjs tags, which are used differently. As I said in the post, the angularjs tag did grow quickly and had the 10th highest rate of year-over-year growth. This is not as fast as those other two, though; the angular and angular-directive tags were both in the top 6.
Roger that. I thought in the writeup you were comparing the growth of angular / angularjs and referring back to the graph (which didn’t have angularjs of course). Now I’m tracking. Thanks!
Interesting analysis, though I don’t know if looking at tags of questions is the best indicator. I think this would likely introduce a bias towards newer languages, frameworks, etc. I hardly ever ask a question on stackoverflow, because most of the questions I do have have already been asked and answered. I think that maybe looking at page views or votes (or perhaps some sort of combined metric of votes, views, and questions) might yield a different story. Although perhaps that is not the story you are trying to tell.
Exactly! I would also assume that there are more question when documentation is poorer, so this can all be attributed to changes in documentation!
This is a question we get a lot! We do look at traffic for quite a number of our analyses and find that in almost all situations, trends in questions asked track with traffic trends very closely. We do not see technologies that get a lot of traffic but not a lot of questions being asked. This is *especially* true for growing technologies, which is exactly what I was looking for here.
Where we do sometimes see traffic and question trends diverging a little bit is for very mature technologies, but these are not fast-growing technologies and not what I was looking to analyze here.
Thanks Julia! Out of curiosity, do you have a correlation coefficient you could report between questions and traffic?
Not without context (which years? which technologies?) but that’s a pretty great idea for a blog post.
“most of the questions I do have have already been asked and answered”
I think that alone could be a sign of a language in stagnation.
language stagnation is just be another way of saying language maturity
That is like saying stagnation of intellectual curiosity would be the equivalent of maturity of an individual. What otter wubbish.
yes I agree, your statement is rubbish, but it’s ok equivocation and faulty analogies are common logical fallacies
It was you who claimed the connection stagnation maturity. I just wanted to point out that it is equally brain-dead when comes to language as when talking about individuals stagnation contra maturity.
My statement was in a single domain, languages. It is a faulty analogy to compare the life cycle of property of an person “intellectual curiosity” to the life cycle of an person a whole. In fact if your argument is adjusted to remove the imparity, stagnation of an individual as maturity of individual, it actually stands as a description of the aging process.
Have you heard about a metaphor? A language not willing to change stagnates. An individual not willing to change stagnates. You can choose to call either of those “maturing” but it is not true in either case. It is like pretty painting a pig. It is still a pig. Maturing means something else. What you intend to convey seems to be someone shifting from a growth mindset to a fixed mindset. There do indeed exist old people still in a growth mindset as well as old languages willing to develop and grow, thank you very much…
Your statement was an analogy not a metaphor. As an analogy it was inaccurate. Metaphors are best left to poetry not discussions of reason. As an example, “pretty painting a pig” is a metaphor. A language does not have a “will” and therefore does not have a will to change nor a will to not change, therefore your given statement is false. Your subsequent statement that it is “true in either case” is unsupported by reason, since the prior statement is based on an assertion that is false. I intended to convey no such meaning as you have asserted, since I was not speaking of people. However, when speaking of people, as biological entities, yes cellular stagnation begins at cellular maturity. I also chose to call nothing “maturing”, I never used that word. Had I used that word it would have been in reference to something that had not yet reached maturity, since something that is still maturing by definition cannot have reached maturity. An entity in this stage is still in growth. It is only after growth subsides that maturity is reached. Once maturity is reached stagnation begins. I agree with the first half of your last sentence, there are old people still in a growth mindset, however we have been discussing languages not the mindset of people. I must also again disagree that languages have a will.
I suspect the arrival of one skilled and motivated expert on the StackOverflow scene can make a big difference to the number of questions asked. How much is the growth or decline of SO activity for a technology related to the technology itself, and how much to the good functioning of the SO forum for that technology? It would be interesting to see analysis of purely internal factors, e.g. whether growth in the number of questions is correlated with the number and quality of answers.
Not really easy to measure but makes me wonder if the documentation lacks or there are non-intuitive “features” in a technology leading to more questions, especially for beginners.
Angular is hard to use compared to VueJS for instance. Thus more questions.
This blog post stinks of simple-minded frequentism. As a Bayesian I am appalled. I see a glimmer of hope however in statements like “…Note that AngularJS also grew incredibly fast…”. Yes, yes, you really do want compare the thing your studying to an alternative. Keep thinking along those lines and soon enough the glory of Bayes will be revealed to you.
“stinks of simple-minded frequentism” — yes …. “Data Scientist” — ?
“Backbone.js is a JavaScript framework, with the brutal lifecycle typical for that ecosystem” What does that mean? I haven’t heard that sentiment before.
Javascript frameworks come and go like farts in the wind
I am curious, where does R or python line up on the graph? What the other smaller stack exchanges like about the data science/cross validated stack exchange?
^My point: I would be curious how stuff in the middle of the pack compares with the top trends. (interactive chart prehaps?)
Also I never new people were answer questions about tensorflow on stack. That is pretty cool! 🙂 Do you know if any of them are teaching it? (<==I been searching for person to help me learn tensorflow for over 6-8 months.) Either way, now I new category to look forward to on stack-overflow.
You can check out some analysis including Python and R here: https://stackoverflow.blog/2017/09/06/incredible-growth-python/
They are growing quickly, but not with the kind of super dramatic shifts from this post.
Probably the most captivating article heading I’ve seen in a while. However I would agree with most of the sentiments here. This graphs don’t exactly represent “shifts”. A nose-dive in the number of “questions (per month)” for these categories does *not* in of itself indicate a “shift”. Keep in mind that Stack Overflow has a specific purpose which is to share knowledge and when its fulfilling that purpose very well and consistently for any of these technologies *and* that tech becomes stable, then of course we expect to see the questions decline. Why? Because all the questions are asked and the answers are all there. Nothing much is changing with the tech. Doesn’t mean ppl are shifting from it, it could just mean all the knowledge is now out there. What you’ve failed to do is correlate the page visits to these questions. Sure, new questions tagged “x” may have nose dived, but maybe they have more page visits than tech “y”. Show me the money.
This is how you find out whats dead and whats not: put out a job posting for a remote programmer who can work what whatever hours and get a great salary working on start-up _______. Then ask them what tech they propose to use for ________ and ________ and why. Also ask them what they would not use and why. Go through the stack of resume’s and digest that.
“we’ve talked recently about technologies that are shrinking in popularity and those that are growing” – both links take us to the same article, is this expected?
Thanks for the note; fixed now.
Are we sure we are measuring popularity by analysing number of questions asked? Seems to me we’re sort of measuring difficulty of use, too. (It does not surprise me that Angular generates lots of questions!)
I had this same thought. I do think that a large volume of questions indicates developer investment in a technology, yet firsthand experience tells me there’s a real possibility some technologies are more complicated (and harder to debug). Also, they may benefit from a “me too” herd mentality. That said, anything has to be at least reasonably good to get traction. It will also benefit if (unlike Parse) there’s a commitment from the originator. I was greatly relieved, for example, when a Google engineer assured me they use Protobuf for everything, having invested significant resources into learning how to use it.
Or a failure of technology “owners” to provide decent/thorough documentation elsewhere?
Is vim really so ridiculously popular it generated a million views?
Not at all – it’s just so horribly unintuitive that people can’t figure out how to use the thing.
Lots of “Angular” questions I’ve come here to look up were a direct result of Angular & Ionic having incomplete/outdated docs and dead-silent forums.
While I’m not surprised in the rapid growth of Swift & Angular, the rise of some topics over others also speaks to the usefulness of the platform vs. alternatives.
This is another question we get a lot! We look at traffic to questions for quite a number of our analyses and find that in almost all situations, trends in questions asked track with traffic trends very closely. We also look at how traffic per visitor varies across industry and company and find that it does not vary much. All our analysis indicates that developers use Stack Overflow when they are working and learning, that our traffic data (and our Q&A data) is a good tracer of developer activity.
Really fascinating look at the technology landscape for growing and shrinking tech. Thank you for the compilation and analysis!
great post thanks for sharing the research
Don’t forget that if we don’t count duplicates / closed questions from tags, old technologies (like ORM) tends to have a so huge pool of questions/answers that any problem encountered have already been asked and answer on SO.
We can see that parse.com & backbone are example of technologies that are growned as fast than they shrinked.
While if we took matures technologies, we can hope than quite a lot of the new questions (not closed) are about the last functionalities that they brought while the old one have already been answered since quite some times.
As such I’am not sure than if we talk about long-living technologies, SO is a very good indicator of their popularity. Eventually they can be a good indicator if those technologies still bring a lot of new stuff (an so new questions, C++ might be a good example of that) or [ don’t or eventually is really being less used, C might be a good canditate for that ].
ORM is a description of a programming technique, unlike all of your other terms which are specific technologies. Picking 2 specific ones, Hibernate and Django, they seem to have increased relative to SQL:
https://insights.stackoverflow.com/trends?tags=django%2Chibernate%2Csql