Why is Python Growing So Quickly?
We recently showed that, based on Stack Overflow question visits, Python has a claim to being the fastest-growing major programming language, and that it has become the most visited tag on Stack Overflow within high-income countries.
Why is Python growing so fast? Python is used in a variety of purposes, ranging from web development to data science to DevOps, and it’s worth understanding what particular applications of Python have recently become more common. I’m a data scientist who uses R, so I’m certainly interested in how much of Python’s growth has been within my own field. In this post, I’ll take another look at Stack Overflow data to understand what kinds of Python development have been growing, and in what kinds of companies and organizations it’s most used.
These analyses suggest two conclusions. First, the fastest-growing use of Python is for data science, machine learning and academic research. This is particularly visible in the growth of the pandas package, which is the fastest-growing Python-related tag on the site. As for which industries are using Python, we found that it is more visited in a few industries, such as electronics, manufacturing, software, government, and especially universities. However, Python’s growth is spread pretty evenly across industries. In combination this tells a story of data science and machine learning becoming more common in many types of companies, and Python becoming a common choice for that purpose.
Just like in the previous post, all of these analyses are constrained to World Bank high-income countries.
Types of Python Development
Python is a versatile language used for a variety of tasks, such as web development and data science. How could we disentangle Python’s recent growth across these fields?
For starters, we could examine the growth in traffic to tags representing notable Python packages in each field. We could compare the web frameworks Django and Flask to the data science packages NumPy, matplotlib, and pandas. (You can also use Stack Overflow Trends to compare rates of questions asked, rather than ones visited).
In terms of Stack Overflow traffic from high-income countries, pandas is clearly the fastest growing Python package: it had barely been introduced in 2011 but now makes up almost 1% of Stack Overflow question views. Questions about numpy and matplotlib have also grown in their share of visits over time. In contrast, traffic to Django questions has stayed fairly steady during that time, and while Flask is growing it remains at a smaller share. This suggests that much of Python’s growth may be due to data science, rather than to web development.
However, this gives us only part of the picture, since it can measure only widely used Python-specific packages. Python is also popular among system administrators and DevOps engineers, who might visit Linux, Bash, and Docker questions alongside Python questions. Similarly, plenty of Python web development is done without Django or Flask, and such developers would likely visit JavaScript, HTML and CSS as “supporting” tags. We can’t simply measure the growth of tags like linux, bash, javascript and assume they’re associated with Python. Thus, we’d like to measure the tags visited alongside Python.
We’ll consider only visits in the summer (June-August) of 2017, which helps reduce the effect of undergraduate students, focuses on recent traffic, and helps reduce the computational challenges of summarizing traffic across long periods of time. We considered only signed-in users who had visited at least 50 Stack Overflow questions during that time. We considered someone a Python user only if a) their most visited tag is Python, and b) Python makes up at least 20% of their visits.
Which tags were often visited by the same people who tended to visit Python?
Pandas is by a large margin the tag most visited by Python developers, which isn’t surprising after we saw its earlier growth. The second most visited tag by Python visitors is JavaScript, which likely represents the set of Python web developers (as does Django a few slots lower). This confirms our suspicion that we should consider what tags are visited alongside Python, and not just the growth of Python-related tags in general.
Going down the list, we can see other “clusters” of technologies. We can examine their relationships by considering what pairs of tags tend to be correlated: that is, whether pairs of Python users are disproportionately likely to visit both tags. By filtering for pairs of tags with a high Pearson correlation, we can display these relationships in a network diagram (see here for more on this kind of visualization).
We can see a few large clusters of technologies, which roughly describe categories of problems that are often solved with Python. In the upper center we see a cluster for data science and machine learning: it has pandas, NumPy, and matplotlib at the center, and is closely connected to technologies like R, Keras, and TensorFlow. The cluster below describes web development, with tags like JavaScript, HTML, CSS, Django, Flask and JQuery. Two other clusters we can spot are system administration/DevOps on the left (centered around Linux and Bash), and data engineering on the right (Spark, Hadoop, and Scala).
Growth by topic
We’ve seen how Python-related Stack Overflow traffic can generally be divided into a few topics. This lets us examine which of the topics is responsible for most of Python’s growth in Stack Overflow visits.
Imagine we were looking at the history of a user, and we see that Python is their most visited tag. How might we guess whether they are a web developer, data scientist, system administrator, or something else? Well, we could consider their second most visited tag, then their third, and work our way down the list of their most visited tags until we saw something recognizable from one of the clusters above.
Thus, we propose the following simple approach for classifying a user into a topic, where we find the tag most visited by each user from the nine listed below, and use that to classify them.
- Data scientist: Pandas, NumPy, or Matplotlib
- Web developer: JavaScript, Django, HTML
- Sysadmin/DevOps: Linux, Bash, or Windows.
- None: None of the nine tags above made up more than 5% of their traffic.
This isn’t very sophisticated, but it lets us quickly estimate the influence of each major category on Python’s growth. We also tried the more rigorous approach of latent Dirichlet allocation, and got qualitatively similar results.
Which categories of Python developer have become more common over time? Note that since we’re categorizing users rather than question visits, we’re showing this as a percentage of Stack Overflow registered visitors (whether they visited Python or not).
We can see that the number of Python visitors who work with web technologies or system administration is growing at a slow or moderate pace in the last three years, out of all visitors to Stack Overflow. But the share of Python developers who are visiting data science technologies is growing very rapidly. This suggests that Python’s popularity in data science and machine learning is probably the main driver of its fast growth.
We could also consider growth on the level of individual tags, by calculating the traffic to tags visited by Python developers in 2016 and 2017. For instance, it’s possible that Javascript traffic is steady overall, but that it’s shrinking as a percentage of visits from Python developers. Once we have those per-tag growth rates, it’s useful to lay them out in our network to understand what topics are growing and shrinking.
This helps confirm our suspicion that a lot of the growth within Python is related to data science and machine learning. Most of that cluster is shifted towards orange, meaning those tags have started making up a larger part of the Python ecosystem.
Industry
Another way we can understand the growth of the Python language is by considering what types of companies it is visited from. This is a separate question from the type of developer a visitor is: both retail companies and media firms could employ data scientists or web developers.
We’ll focus on two of the countries in which Python’s growth is most notable: the United States and the United Kingdom. In these countries, we’re able to segment our traffic by industry (just as we did to compare AWS and Azure).
The industry with the greatest amount of Python traffic (by a large margin) is academia, comprised of colleges and universities. Is this because Python is often taught in undergraduate programming classes?
Partially, but not entirely. As we saw in a previous post, Python traffic from universities is common in the summer, not just in the fall and spring. For instance, Python and Java are the most visited tags from universities, and we can see the difference in their seasonal trends.
As a percentage, we can see that traffic to Java drops more sharply during each summer, because Java is a relatively common subject in undergraduate classes. (We’ll be exploring what programming languages are most taught at universities in a future post). In contrast, Python makes up a larger share of each summer’s traffic. The high traffic to Python questions from universities is therefore due partly to academic researchers, who generally work throughout the entire year. This provides more evidence that Python’s growth is due to its capacity for scientific computing and data analysis.
As for the other industries, we’ve already saw that Python is popular and fast-growing in the government sector, but we can see it’s also widely used in the electronics and manufacturing industries. I’m less familiar with those industries and would be interested in insights as to why. The language still hasn’t caught on as much in retail or insurance companies (some investigation shows that Java remains dominant there).
This post is primarily investigating causes of Python’s growth. Was Python traffic growing more quickly in some industries than others?
The growth of Python in the last year has been pretty evenly spread out across industries, at least in the US and UK. In each industry the traffic to Python increased about 2-3 absolute percentage points. (Note that this represents a larger relative growth in industries where it wasn’t already common, such as insurance and retail).
In many of these industries, Java remains the most-visited tag based on 2017 year-to-date traffic, but Python has been making progress. For example, within finance (one of the larger contributors to Stack Overflow traffic out of these industries), Python went from being the fourth most visited tag in 2016 to the second most visited in 2017.
Conclusion
As a data scientist who previously worked in Python but now works in R, should this push me towards switching back?
I don’t think so. For one thing, R has been growing rapidly as well; we saw in the last post that it’s the second-fastest-growing major programming language, after Python. But secondly, the reasons I prefer using R for data analysis aren’t particularly related to its relative popularity. (I’m planning on writing a personal blog post about my own journey from Python to R, what I like about both languages, and why I don’t feel compelled to switch back).
In any case, data science is an exciting and growing field, and there’s plenty of room for multiple languages to thrive. My main conclusion is to encourage developers early in their career to consider building skills in data science. We’ve seen here that it’s among the fastest-growing components of the software development ecosystem, and one that’s become relevant across many industries.
If you work in Python, for web development, data science, or another domain, and are looking to take the next step in your career, here are some companies hiring Python developers right now on Stack Overflow Jobs.
156 Comments
Not at all surprised.
Fantastic Analysis. Do you the code openly published for this analysis?
This is a good one.. I wrote a post about Python. And, the respond showed how big the language is!!
Heres my post if you like to view:
https://medium.com/@amtoddstewart/are-you-interested-in-python-programming-cd0c395b8a3d
You ask why Python is growing in electronics/manufacturing, and as a longtime embedded engineer I can toss in some insight.
There is a constant need for creating test harnesses for testing electronics during development, and Python is slowly growing into that niche. The most common solution has been building full-up C/C++/Java apps, which
requires dedicated resources to write/maintain, compared to using a scripting language. With a scripting language, a hardware engineer can write quick scripts based on an I/O library.
The original common scripting language in that space was Tcl, and that’s still the scripting language built into some of the big CAD tools for tasks like chip/board layout and circuit simulation. But Tcl is a quirky language that doesn’t have the mindshare that Python does, and Python is becoming the language of choice for test harnessing in that space.
As an electrical engineer, this is true. It is so much cheaper and faster to automate testing with Python than something expensive and inflexible like Labview.
Also serial interfaces, CAN buses and others tools like power supplies and such are dead simple to get up and running for various controller test setups.
The embedded data itself is also something that is often overlooked and requires some serious tools to parse and visualize at millisecond resolution from hundreds or thousands of sources. Some data sampling can easily rival the amount of data generated by big data companies/apps.
There are also plenty of other day to day uses such as code generation for embedded code bases and pulling data into and out of Excel which so many engineers love to use even if a database may be better suited.
Python is popular in electronics and manufacturing because it is versatile and there is nothing else that can do all of those things.
Really? Weird, why aren’t you guys using the same C interfaces with NodeJS then? NodeJS has a way to interface with C/++ interfaces as built modules that accomplishes the same thing. Heck Beaglebones COME with a prebuilt module for NodeJS called bonescript for interfacing with the serials. We love it!
NodeJS is async = horrible to work.
And a response like that proves you don’t have a clue. Cutting and pasting is what you do when you don’t know how to write code. If you need the same code twice, it should be refactored into a single routine and called in both places. The single best piece of advice that came out of the Extreme Programming movement: Refactor Ruthlessly.
I do not agree
with you, perhaps you who did not understand, the syntax of python
which is based on the blank space and a tabs puts the code
fragile (not necessarily for the copy and paste,
but there are several fact that causes this problem, for example a change of
encoding by error during a change of family of OS very different, …).
I understand Python syntax well; I’ve been writing in it for over a decade.
Python syntax is no more or less fragile than any other language. Have you ever missed copying a parenthesis in Lisp, or a brace in C? It’s at least as bad or worse than forgetting to re-indent something in Python.
And changing encodings shouldn’t affect whitespace. If it did, you were probably mixing tabs and spaces, in which case you’re a fool and deserve what you get. 🙂 Convert to all one or all the other, and conversion isn’t a problem.
You must know what promises(futures), generators, and async/await. It’s simple and easy to do
No, it’s just different. Perhaps you should grow some engineering balls and expand your understanding of things.
Why should they learn async, if it doesn’t provide any advantages for the kind of code that they write, but does make it more complicated.
Not everything is a web app, you know.
Event driven, out-of-loop async processing, generators and futures are usable in every distributed or user-facing environment. Which leaves synchronous linear-executed code to either be on the “inner execution” end of some IoC method (like how Apache executes PHP in web context) or for scripts whose execution the OS schedules.
I’ve written GUI stuff on C# for Windows. If you’re not thinking about how not to block the execution of the UI when handling large jobs you’re writing awful code and I’d hate to be your user. It’s more/less the same across the board. If your code needs to do a lot of stuff automatically, it better not block. Who will handle it (you or whoever is calling you) largely depends on the environment. But you better be thinking about it. Always.
Okay, I’ll grant you that web is not the only niche where async is useful. Desktop UI, too.
That still leaves plenty of code where it’s not particularly useful. Most command line utilities. Most compute stuff. Certainly, most things likely to show up in the environment that we’re talking about in this thread.
You *can* use it in those other cases, yes. But you aren’t actually reaping many benefits (if any at all; having your code rewritten as a bunch of callbacks imposes considerable perf overhead, so you’d better have enough actual asynchrony to be worth it). And you do make things more complicated. The last thing you want around embedded and heavy industrial stuff is “more complicated”.
Of course, if you do really need async, Python has it. In fact, it’s more powerful than in JS (you also get things like “async for”, “async with”, and async sequence comprehensions). But Python doesn’t insist that everything *must* be async – because that’s simply not true.
Python can reasonably call C/C++ modules, and that’s a reasonable way to run things.
Tcl is even better at wrapping C, or being embedded in C for a command line, and has an excellent event driven architecture that allows one to cleanly write code that would require threads in most languages. I prefer it in a number of ways. But it’s hard to find Tcl programmers, and easy to find Python programmers, so for new projects I reluctantly give the nod to Python.
I can’t personally speak to NodeJS, but the JS roots really make me leery of it. Automatic type conversion can really get you in trouble when dealing with embedded systems where the number of bytes a number is stored in is critical information.
Oh, and the C interfaces is the part we’re trying to get away from. I’ve written a number of pure-Python & pure-Tcl serial protocol interfaces, and performance has been excellent. Plus in either, script portability is better than a compiled code block – I can send test scripts to another engineer who runs them on his Mac, and it’s just not an issue.
Special systems(like embedded) are limited by cost, and thus size matters there. JS is by no means small, but it can be run on arduinos. And for larger systems why not just use a beaglebone? L
Cost is the big reason there, and so because of that no matter what in a size constrained environment compiled to machine code programs will always be the only option. Can’t contest that.
You may misunderstand me. The embedded side of what I do is pure C, on tiny processors. JS is completely too large – I have a total of 48K of program memory, and 8K of RAM on my current target, and I’m counting bytes to make things fit. I can’t spare the room for an interpreter for Tcl, JS, or Python.
The scripts are all on the host, and are used to feed inputs into the embedded target processor and look at the outputs. For this, the size is not the issue – developer time is. JS might be a reasonable alternative, but it has no advantages I’m aware of over Python or Tcl. Certainly it has much less existing support libraries for test equipment interfaces and the like.
A perfect response. Support is king, regardless of implementation when time is of the essence.
As for type conversion, there are bitwise operators, sorry for not addressing that earlier. Raw buffers and bitwise operators give a very fine grain control, but if you need to count bytes, python wins there. For implementation speed. Are you using an esp? Those stats sound very familiar, but I may be miss remembering ram size…
It’s an MSP430 for this project, though I’ve used AVRs & PICs on past projects. Most projects here need extraordinary battery life, so the MSP wins on that front, and we stay in the family just to keep tool suite proliferation to a minimum. The MSP430 family has enough range to cover any requirements smaller than a full-up ARM.
Good chip family. Ti hit it out of the park with those
Actually, Python is much, much easier to get into than JavaScript. The syntax is less weird, more English, less odd characters, and, unlike Node which is a bit like Linux really, wild-westish, Python has a pretty standardized library and an even larger ecosystem (tho Node is enormeous as well). Node is really for when you actually need it to work decently fast and need async, event-driven behavior. In that regard it’s much more alike Tcl, except with a saner, C-like syntax,
You’re joking about the more english like syntax right? C like syntax is the correct term here. As far as I know, Haskell, infix operations, and other such ways are the only way to write english like programs. What you’re talking about is composition and lexical arrangement, which is all a design focus and personal choice. Proper libraries can make any code look like english. An example:
take(a, from, b)
You will never be able to say “That is a specific language.” That can literally be any language in which functions are first order objects(anything Turing can do, I can do, etc.). Im just not seeing how python is more english like than other languages. Well except maybe whitespace, cats, and typescript/superscript. Those languages throw a curveball of specifically being logiclike languages instead of english like, regardless of what they compile too.
I’m geniunly curious what you mean by odd characters though. I get confused when looking at python and seeing function (tuple*) or function(derencedObject**) or function(a, b, c)
Most of your comment sounds like personal opinion. Also I’m pretty sure JS core is pretty standardized and published.
I will admits NodeJS is kind of cowboyish, but so is python in that regard. They’re both standard sets of libraries included with the language for that inerpretur, and you have no idea if someone has extended, replaced, or redefined anything anywhere. But that again comes down to design patterns, which isn’t language specific. Any asshole could decide to redefine something somewhere without you knowing because people suck. Thankfully the variable definition of const has made that very hard to do in JS.
I will admit both languages are exceptionally fun to program in. I know python, c/++, Haskell, Microsoft visual basic, JavaScript, java, and php as my competent languages. When I need to do something I always reach for JavaScript or python when no UI is involved. Even then it’s usually JavaScript and HTML(except the rare case I need to do a native gui, then it’s MVB). They’re just so much fun and fast to use with extremely little setup.
In my own opinion, the biggest advantage that Python has over JS, Tcl, Perl, Lua, or any of the other scripting languages is its clarity of syntax (not exactly simplicity, but related). It was designed by a working programmer, not an academic, and its goal was to make correctly written code be as easy to read & review as possible.
That doesn’t matter on small, personal projects. But if you’re part of a team writing test scripts to test the output of a different team’s code controlling hardware designed by yet another team, and everything has to be code reviewed and verified, code clarity and readability is a really big deal.
And despite that, it still manages to be fun to program in. 🙂
And you can program in imperative, object-oriented, or functional modes. This really helps introduce it to people from other fields who are new to programming – they can use the simplest approach to write test scripts, not caring that under the hood of the library we provide them lies complex meta-programming in an event-driven message system with auto-generated mutating objects that have their own stateful lifecycles. 🙂 It’s all the same language, just used at different levels by different engineers each at their own level of capability.
I feel like those are the exact strong suits of both python and js(syntax aside). Considering my code has made it easy enough for a Perl programmer, a python programmer, a php programmer, and a java prigrammer to get to an advanced point within weeks, I feel like both languages are a win.
What i will give pythin is a plethora of recent good practice articles, while JavaScript is just now getting to that point. Learning coding is becoming easier across the board though, regardless of language.
A Perl programmer can cut through anything. Even un-mnenomic machine code 🙂 Jokes aside, ATM I’m a Node/JS programmer primarily, but you cannot seriously claim that it’s equally legible or elegant as Python. The true reason most of us use Node is because it’s almost as fast as, say, Go or Java, and almost as sane and expressive as, say, Python.
> And you can program in imperative, object-oriented, or functional modes.
Which can be said about JavaScript as well. Especiialy ES6+
> You’re joking about the more english like syntax right?
Perhaps, with Python, the proper term would be “legible”. Much less symbol magic than with JavaScript. Jesus, how can someone put something cryptic and illegible as Haskell and “English-like” in one sentence. It wins in the “which one of us looks more like something you’d see on a blackboard of undergrad calculus class” hands-down. And I’m an engineer by eductaion so not afraid of symbols.
> I’m geniunly curious what you mean by odd characters though
I.e. char not in [a-zA-Z0-9]. I’m a Node programmer mostly, now mostly coding with Babel on both the backend and frontend tho evergreens and Node 8+ support pretty much everything I need from ES6+ so this truly is the state of JS today.
In JavaScript I tend to need to write stuff like:
myFun: (external, parameters) => (internal) =>
internal.filter(x => parameteres.every(
part => (x != part && !part.includes(x)
)
Now, thank God for ES5/6/7 or it would be a nightmare of nested for loops just a few years ago.
Something like that would be more verbose and a nag, but simply more legible in Python
def myFun(external, parameters):
def internFun(internal):
return filter(
lambda x: x not in parameters and
not any(x == y[1] for y in parameters.values())
)
return internFun
Both are contrived, and off the top of my head so pardon for syntax/logic/idiotic errors.
Now, I’m not claiming latter is mother of legibility but it certainly imposes much less mental strain to parse the text, so that you’re left with more energy to parse the syntax, thus more energy to parse the algorithm and thus more energy to parse the intent or find that damn bug.
To us your example makes sense in either language, but when I put on beginner glasses neither makes sense. We’ve gotten spoiled and lazy in either case with functions that reference an internal loop mechanism.
Either way both methods require the programmer to know about the internal libraries of the language as they apply to collections. Otherwise this is a bunch of gibberish in it’s own right because it’s extracted out those loops into generic loops by using the fact the language is a first order language and can pass around functions.
Now legible code is the other part I want to look at here:
“`
const keysOf = (object) => Object.keys(object)
const valuesOf = (object) => Object.values(object)
const zipListOf = (object) => [ keysOf(object), valuesOf(object) ]
zipListOf( { test : true } ) // [ [ “test” ], [ true ] ]
“`
Legible code is the developer’s prerogative in how they apply the libraries available to them. The methods `keysOf` and `valuesOf` are both just wrappers to internal libraries to make code more legible for people who LIKE that sort of code.
Some people prefer their transformations to be `piped` or `composed` so as to read either in order of operations or order of composition. That’s their prerogative in legible and easy to understand code. Taking a look at `Ramda` we can see this style all over when they start to realize that programming symbols means these sorts of things are completely doable. Math focused people LOVE that:
“`
import { compose, pipe } from ‘Ramda’
import { f, g } from ‘someLibraryYouCoded’
const compo = compose(f,g)
const read = pipe(f,g)
compo(x) // f( g( x ) )
read(x) // g( f( x ) )
“`
Anyone who is worth their weight in Math will recognize compose. People who prefer order of operations will love pipe. Both can be written in terms of the other. Both can be written in Python and JS. Both are EXTREMELY useful for applying chains of transformations.
Often times what is legible to one programmer, is complete nonsense and bad practices to another programmer. Such is the hell of the programmer who witnesses other’s.
I will concede it’s always a headache when someone does the following though:
“`
// ACTUALLY OKAY IN JAVASCRIPT
{ ‘u1300’ : true }
“`
However that’s because MOST real strings in javascript are okay as keys in Objects. Of course even JS has it’s list of things you can’t do as a key, but it is very liberal so you can do REALLY stupid things like
“`
// admittedly this has very few uses… but it’s fun!
compose(zipObj, reverse, zipListOf)(o)
“`
Which could produce some very interesting things. It also allows more single byte keys than you might think which could allow for some EXTREME compression of JSON by converting the strings to Unicode characters to save space, and decompressing them on the other end. For the same functionality in Python you would need to build a converter. Not sure that’s a good thing though since obfuscation isn’t secure, or even helpful in most cases… But boy it’s fun!
Either way both languages are both just as usable by experienced developers and even end up looking the same in the end. So I’m not convinced on the `python is more legible` opinion since they’re both programming languages and they aren’t meant to be human legible. That’s the developers job! I will give that things can be less confusing because of the key-space javascript allows.
It looks like you have a Javascript hammer and that you see nails everywhere.
Coincidentally most of my exposure to Tcl has been through Python by programming user interfaces with its bundled Tkinter.
Unfortunately, that’s true of a lot of folks.
I say unfortunately because I think Tcl is an underappreciated language. But the part that’s exposed through Tk doesn’t show it to advantage, because Tk adds its own layer of quirks on top of the simplicity of basic Tcl.
Yeah, it was tough but rewarding programming tiny games (minesweeper etc.) in it.
Python is certainly increasing in popularity and adoption.
Another insight from Stackoverflow’s Survey 2017 of “Most Loved, Dreaded, and Wanted Languages”, (https://insights.stackoverflow.com/survey/2017): Python is the ‘most wanted’, but it’s ‘not’ the most loved, while ‘Django’ is completely absent.
Quickly?
Then why is python3 has been out for almost a decade yet so many tutorial and libraries still stucj in python2?
Debian.
Python 3 is easy to use in Debian, it’s just “python3”.
Well, the whole Debian still relies on python 2.7 for its system tools.
Libraries aren’t stuck on Python2 anymore. Essentially any library of consequence has been python 3 compatible for some time now.
http://py3readiness.org/
345 of the top 360 libraries in the PyPI repository support Python 3.
I am aware of Python2 being used for system purposes in Linux. Python2 was used to build various system programs early on, once built and working nobody wants to upgrade them to Python3 unless it is absolutely critical because there are better things to do with time. So the system default python of Linux Ubuntu 16.04 (the latest long term stable) is still Python2. Python2 has a smaller footprint as well, so it is better suited as a system language for smaller systems.
But Python3 is always used for new programs.
This is actually a grave misunderstanding. Due to problems that were found in Arch Linux when `/usr/bin/python` was referring to Python 3, PEP 394 was written, and it pretty much says that `/usr/bin/python` should refer to Python 2 until it dies.
However, if you do a clean server installation of Ubuntu 16.04, it doesn’t have */usr/bin/python* installed *at all*; */usr/bin/python3* however is – and it is an essential system component.
I’m curious if many of the people using Python use it for it’s ability to easily interface with C/++ not knowing that such a layer can be built into NodeJS(javascript) as well in a module?
I’m actually seeing a lot of prominent NodeJS engineers working with C/++ engineers to get programs interfacing with hardware in Javascript, but in all reality that also means the C/++ and JS engineer are actually fairly advanced, causing them to visit SO less often than an academic user would who has no idea NodeJS can do that.
This really suggests to me that people using Python are learning Academics since the big surge is “Coding is the future” compared to how many people were learning JS because “Wen apps are the future”. This feels like a stupid fad that when people actually do research will realize that most languages have the tools and abilities to do what most other languages can…
Any language that is Turing complete can do what another Turing complete language can do. That doesn’t mean they can do it as easily.
I’m curious if you realised that JS design is flawed, that the whole language has been designed in 10 days and that Python syntax is much nicer than JS?
I’m curious if you realize personal opinions matter little and only make you come off like a stuck up? Literally all you said is “I have an opinion”.
That said, JavaScript and python syntax mean little in implementation. It sounds to me like you don’t write to many libraries or actual routines if you think a syntax is better than another. Eventually programming becomes similar to writing stories. Consistent design practices, proper documentation, and good function names means any language can look like any other.
take(a, from, b)
I used this example earlier. You can’t say what language this is. You can tell it will take something from something else using a function to be called.
pipe(take(a, from), take(b, from), take(c, from))(d)
pathWith([a,b,c], from)(d)
What do these do? It should be pretty obvious that they return a function applying the same order of operations, one being a shorthand reducer for the collection, while the other applies functions in order supplied.
Again proper design means syntax means little. The fact that your tuples use parens versus mine using brackets means nothing. A proper language allows you to define your process and implement it.
“Literally only opinion”? No. Javascript has been designed in 10 days, and that’s a fact. The biggest flaw of Javascript IMHO is that it almost never raises an exception, just happily and implicitely converting types into one another in surprising manners (e.g. 1 + ‘2’ and 1 -‘2’). This behaviour makes it possible for JS newcomers to write code that kinda works but is sloppy and brittle. Python (and many other, saner languages) simply complains if you’re trying to calculate 1 + ‘2’. Should it be ’12’ or 3? The language shouldn’t make an arbitrary decision.
There is Laputa, where the most experts use python, RoR, Matlab and many other stuff that are lectured in many universities.
And here in earth, we use Java, C#, PHP, Excel,C,C++ and Sql.
BTW: Its not important the popularity of the language but what you can do with it.
The more popular the language, the more you can do with it (in terms of open source libraries, job opportunities, seminars, books, magazines, online courses, etc.).
This is a completely anecdotal comment. I’m a mathematician by training, a programmer in my consulting life, and a biomedical scientist in my day-job. Biology and the biomedical sciences are among the sciences that have become computational in the last decade or two (which is essentially how I ended up here) and as such I’ve found myself having to train Ph.D.-level scientists, in their 20s with no programming experience, to program. I’m a Java programmer, but Java is not suitable for new programmers imho, R is just a bit too strange unless you’re a statistician (I mean, I get why you would make all data types vectors for that purpose, but for a new programmer, no thanks); Python just hits that sweet spot of being amenable to new programmers while being powerful enough to do real science. So I teach my biologists Python, and they pretty much like it. My guess is that’s not unique: as computational science requirements bleed out into all the other sciences, it’s just at the right level of accessibility for the people who need it to learn.
If you’re doing statistical analysis (including data mining and machine learning), R is a powerful language. I don’t think Python approaches the sheer number of packages R offers for data analysis and graphs. And it’s free!
Yes, of course, and we do use R for those kinds of applications. But R is pretty esoteric if you’re in your mid-20s and you’ve never programmed a computer before*; I’d argue Python is a much easier route into programming. I’d rather teach someone who’s getting started with tasks such as reading a genome file (plain text) and finding all occurrences of a simple motif, than running ridge regression or lasso on a large and complex data set. Of course we need to do both, but one is more amenable than the other; I suspect a lot of new programmers are getting their *first* exposure to programming with Python. (*The fact that there are so many scientists in this category is whole other issue…)
Sure. R is a powerful, great language. If you learn R, you cannot apply your knowledge for anything much different than statistical analysis. If you learn Python, you can write games or web services with the same syntax.
The advantage is that your biologists can now write web-services, linux scripts or games with the exact same syntax.
It could be a better-fly effect. One guy sees another guy using Python, so he does. Another guy sees him using Pythons, so he does… This continues until we think the world is converting into Python programmers.
You misunderstand the butterfly effect. The butterfly effect demonstrates chaos. So in this case it would be one guy sees another guy using python… Kim Jong Un spills kimchi on his laptop… my cat learns lisp.
I think ‘the snowball effect’ might be a better fit for metaphors.
Lets not just gloss over the fact that your cat is learning lisp.
Good point. But I do not think I did. Programming is chaotic in of itself. So each guy that learns Python gets a sufficient amount of chaos to satisfy anyone that wants it in the butterfly effect.
You do have a point : Python achieved some critical mass. It’s used in enough places, with enough distinct use cases and with good enough code that there’s no reason to switch to another language.
I think the biggest reason for Python’s uptake is that it’s the dominant language now taught in 1st year University in the high income countries. A few years ago computer science departments switched from Java to Python. They did the same trick years before switching from Pascal to Java. Before that it was Fortran.
And before it gets replaced it well be preceded by JavaScript in elementary school.
We can only hope :).
I think one of the main reasons why Python is getting much more widely used recently is that it is more approachable? to people than compared to, say C-family languages (no scary curly brackets and semicolons, etc.)
Indentation-sensitivity is scarier to some.
It sure is, to me. 🙂
Makefiles induced that in me. Of course, Python is much better when it comes to this but still.
It’s mostly scary for people coming from other languages. For complete newbies to programming, it makes more sense than curly braces, if anything.
Probably, yes. The syntax is nice, it’s quite concise, it’s relatively fast and is used in many different places. Clever people at Google/NASA/MIT keep improving the libraries. It’s free and libre. With the same syntax and language, you can replace stuff you’d do with Matlab/R/Java/Rails..
A lot of folks have been asking this, but python has been ‘approachable’ for a very long time. The sudden surge can only really be explained by some other factor, and I think the SO team made a pretty good case for the machine-learning libraries being the major factor
Hi David, Great Post!!
A bit of advice i would like to have from you.
I am a java developer and i have just started learning hadoop, than will learn Spark and will be looking to shift on these sides. Am i on the right track? Or what you would advice me to do?
Sincerely scared by this. The idea of being one day forced to write code in a language where the deletion of a single white space causes a total change of the meaning of the code is seriously scary.
I hope you guys all redeem one day, see the light, and stop this suicidal trend!
( I am of course joking 😀 )
a single white space is just a character, ascii 32, isn’t it?
I wouldn’t delete random characters in a C program either …
I’m joking as well 😉
But I used to hate Python for that and now that I’ve changed my mind and I’m in love with Python I find that to be somehow a great idea …
Maybe, some day, I will be forced (with a gun pointed at my head) to try and end up agreeing with you… 🙂
Except that unlike Python with that deleted space which would blurp out “IndentationError”, C would likely compile, get deployed, and then wreck havoc.
yet… it’s not a space, it’s a character: ascii 32!!! In Python you can delete all the characters you want, as long as they are following an hash (#) sign… 😀
Just joking, obviously.
I think this point is meaningless, it’s just another syntax style that can be appreciated or not, that’s all. Personally, I like this syntax.
We have been moving toward this coding style for some time now in strongly typed languages as well. For example, the use of var in C# is forced by many code style analyzers and it allows your variables to accept different types of object without changing everything.
For example:
var result = getResult();
Whatever type ‘getResult()’ returns will determine what ‘var’ actually means. This has the same repercussions as Python still ‘working’ after making accidental changes or not refactoring code and accounting for errors.
It’s not like Python doesn’t allow type checks and verifications either. You can write in whatever you want to verify that values and program states are accurate as you go. You’re just not forced to do so.
Python is strongly typed too. You cannot calculate “1 + ‘3’” for example. Did you mean “statically typed”?
C# var and python variables are not the same. C# ‘var’ types are determined at compile time while python won’t know until runtime
Having programmed in Python for over a decade, it turns out that this isn’t an issue.
I guess so. It probably is scaring only at the beginning, like bungee jumping… or heroine 🙂
haha… “or heroine”… WHAT?!
Javascript is the future 😉
Sounds like a slow future….
Sounds like a future as fast as any other stateful language to me… the garbage collection can cause slowdown, but proper temporary scopes solves that, and so do other high performance programming patterns, but that all falls under optiomization. Im just happy my string functions dont cause overflows(im looking at you C…). Ill gladly sacrefice some speed for that…
You can do great stuff with Javascript, but its design is really flawed. Python has some gotchas here and there but it’s a much more mature language and it’s much cleaner than JS. I’d love to see Python replacing JS in browsers but that’s probably not gonna happen.
I always say that JavaScript is the best programming language designed in 2 weeks.
True, but only in the way that C was the future, then Java was the future, then Ruby. Javascript is the most entrenched language in one of the recently hot spaces in programming: front-end web development.
All these languages share one characteristic: they were used to write the “coolest” things of their day, so they became “cool” languages and everyone tried to use them for everything, whether or not it made sense. Eventually, each language settled in its niche, and when another niche got hot the “cool” language of the day changed.
The more things change, the more they stay the same. 🙂
I hope JavaScript can stop to spread like a cancer and just do the job which the language was initially designed to (front-end web development). But probably is just personal taste. We have a lot of technologies, why just not use the best for each purpose? It’s really so cumbersome learning something new?
Your interpretation that visits to Stack Overflow correlate directly to the growth or importance of a programming language seems unintuitive to me. People come to SO mainly because they have problems with the use of some programming language or want to learn it. So your numbers mean first of all that people have increasingly more problems with the use of Python or that more people want to lean it. The first would speak for more inherent problems in Python than in other programming languages, the second might reflect a vast variety of things, among them: Python more used in undergrad courses (because not so difficult) but later abandoned by those who were forced to learn it. I mean, people learn Latin in school, but that does not reflect how important Latin is on the job market.
Your comment seems highly biased to me, and backed by no data at all. A developer never stops learning a language ; why reduce SO to a learning tool for undergrads ?
“The first would speak for more inherent problems in Python than in other programming languages” : you must not like Python much, but correlating the problems of a language with its traffic on SO is ridiculous. SO traffic is a measure of the language USE ; no more, no less. If a language is cumbersome, people tend to not use it less, not to use SO more to debug it.
I think that your answer is neither backed by any data.
TwentyFour has a good point, and what he’s saying is just Common sense, go learn some statistics bro….
“Just common sense” == “I didn’t want to spend time thinking”
in almost all contexts.
And telling someone to use statistics on something YOU said had no data? Really? That ain’t common sense, “bro”.
waw, for me statistics was more than “just common sense”, but maybe you could teach me a thing or two about it, “bro”…
As one of those “privileged” people that comes across nifty tools and programs and then adapts them to things I’m doing, I “stop learning” many languages as soon as my particular project is complete. … or I give up because I no longer care about what the tool or program does. As soon as I no longer care about an end (the results of the tool/program) I stop caring about the means to said end (the programming language).
I remember looking at Rebol and Ruby with glee and thought that they could be game changers as far as effect and clarity, and then real life intruded again and I just didn’t have time to invest in further study.
I suppose that you COULD say that I never stop learning a language because when I find threads of interest about how to do this, that, or the other thing using [pick a language], I’ll look at the thread, and in spite of the thread being out of my depth, I’ll try to see what’s going on and formulate an algorithmic concept for it that maybe I can use later. Maybe that qualifies (to you) as [continuing to learn] a language. I don’t. I won’t remember the case (language specifically) in 6 hours, 6 days, 6 months or 6 years. But, I might remember the methodology and theory of the problem and solution years later and be able to turn it into [insert not-yet-created language here] for some stupid task.
This is one reason I use pastebin or similar and that when I write code, I will comment the everloving crap out of it so that when I come back to it and can’t remember what it is supposed to be doing, that I will be able to hold my hand and remind myself what the code is doing and why. Hell, I’ll even leave special flavor and example information in the code so I’ll know what the code should be expecting and why and what it should be doing with the things (if any) that are supplied to it.
Maybe because I’m doing so many small projects, which as a cursory user of many languages, happens often, I feel like so much of time spent coding is consumed in figuring out [what is it supposed to do?] and [why is it not doing what it is supposed to do?] or [how do I make it do this other thing that seems like something that it could be retrofitted to do?] and that academic study of the full language syntax, libraries, blah, blah, blah, and then a tiny part is making the modifications to make the thing do what it wasn’t doing before.
For those with IT jobs, this concept may seem foreign. But, for many like me (and I’ve been using computers since 1984), this is a pretty normal use case.
A few posts back they looked into exactly what you’re suggesting: https://stackoverflow.blog/2017/08/01/flash-dead-technologies-might-next/
(Skip down to “Conclusion: do questions really represent the health of a technology?” if you don’t want to read the whole thing.)
Insert “THANK YOU!” meme.
Thank you Again .gif
Yeah I don’t agree with this. Walk around _any_ of the startups that I work at, including my most recent one that serviced more than one billion accounts, and you’ll see SO up on lots of screens. We’re in the age of the reference manual, nobody memorizes how languages are used any more – you can’t, you have too many technologies to learn and too little time. Using SO to quickly answer a question doesn’t make you an undergrad, it makes you resourceful.
This wasn’t the point Twentyfour was trying to make. The idea of Python being introduced to students and then discarded was just an example. The point he was trying to make is that SO questions on a particular language don’t necessarily correlate with the use of that language in industry. Only way to know for sure would be to sample the actual data i.e. determine what companies are actually using and present that info. I think there was actually a blog on this subject.
I’d have to disagree with several of your assumptions. Python has not historically been a teaching language; that’s fairly recent and it’s not yet dominant in that niche. And going to SO is definitely not just for language learners; if that’s the only reason you’re using SO, you’re doing it wrong. 🙂
And the Real WTF here is criticizing the SO blog for using SO data to try to gain knowledge of the users of SO. I strongly suspect they know more about their users than you do.
Speaking from a university point of view I can attest to the shift towards Python. On my university R has never been big, and MATLAB has been a mandatory part of most undergrad programs. Since a few years however, undergrads in certain fields receive Python lessons as opposed to MATLAB lessons. When I asked the dean about that he said that in terms of research potential the two are comparable for our faculty, but Python is free whereas MATLAB costs a ton of money in terms of licences each year. Therefor the university as a whole is slowly shifting towards Python, all due to the costs.
“Why is python growing so quickly?”
I dunno, but you better get out of there and call pest control, before it swallows you whole.
Every year somebody comes out with a “new” and “better” language, I seen thousands of them over my 40 year career. The standards are the languages that last. COBOL, C, FORTRAN, JAVA, C++.
Did ADA replace COBOL? Even though ADA was object oriented, multi-tasking, spawning, and could control relays, sensors, and solenoids, plus congress passed a law replacing all COBOL systems in government with ADA, at the last minute the programmers revolted and we still have COBOL today.
When I was finishing college they taught BASIC and Assembly language as the first computer languages kids were exposed to. As I graduated, all the colleges dropped BASIC in favor of PASCAL (the magical language for learning good coding). PASCAL lasted about 4 years.
RPG was supposed to replace COBOL with a “fill-in-the-blanks” form based coding for super duper easy coding – even if you weren;t a programmer. It also failed.
Back in the day every language had a COMPILER that created pure machine code that could be executed at light speed. Now .NET and JAVA rely on a (one step above an interpreter) RUNTIME MODULE, intermediate language and bytecode.
Python is just the latest beginners language served up at community colleges. It has a foreign following, but in the US it’s just a teaching language. Things like Cloture are hot fads – passing code through functions – not new – we used to call it Lambda expressions. Rust, and other browser language are hot topics, along with frameworks, but they are not true languages.
The more things change, the more they stay the same.
I think you may be getting a little jaded.
We use python all over the place at work, and when I was in university (11 years ago) we used python for all of our simulations/computation in the Physics department.
It isn’t really a “new” language any more, and it does see quite a bit of use all around.
Eve Online has been running on stackless python for their entire existence. Here’s a slideshow from 10 years ago! https://www.slideshare.net/Arbow/stackless-python-in-eve
I agree with this. As a front-end developer I’ve been hit time and time again with how Javascript is not a programming language, it’ll never be widely used/accepted, and yet in 2017 it’s closer to being ubiquitous than ever before. What’s the language running all those services that I use all the time? Python. And in fact, in my latest round of interviews I’ve done, I’m hearing more and more of this “We’re moving from .net to Python or Go”.
I wonder how much of this is about accessibility? Python is easier to pick up, and now that the Internet is full of so much good practice writing vs how it was 8 or 10 years ago, its easier to become a legit engineer on your own rather than a ‘cowboy coder’, which increases the legitimacy of Python. Python picked up in popularity after the age of PHP which was basically the poor man’s .net, which due to so few resources that were available to beginners led to the proliferation of crappy code and therefore a bad reputation for PHP. Python’s community has coalesced around better code and hasn’t suffered from the same speed bumps as PHP did.
In short, don’t do what everyone did with us webdevs and deny that Python will become something powerful. It already is, and to ignore it is to do what all those guys at Microsoft I worked with did — eventually you fall behind and become irrelevant. This isn’t a dig at Java/C/.net/etc, but to ignore what the industry is actually doing is to shoot yourself in the foot!
Oh dear. Time has certainly made a fool of you, hasnt it. Where to begin with this trainwreck of a comment.
Well for one thing Java (or JAVA as you adorably call it) is actually 4 year younger than Python.
Python powers in large part, youtube, google, dropbox, survey monkey, quora, bitly, reddit, pinterest. Remind me, what websites are being built using COBOL and FORTRAN? 🙂
Maybe time for your nap eh? 🙂
When you grow up I’ll bet you’ll work for FoxNews. Oh wait, they hate Indians and Mexicans.
Oh come on now old timer, if you were at all able to follow the theme of this conversation, you would have been able to guess that I already have a perfectly good career. As a Python programmer. Or P.Y.T.H.O.N. as you might prefer to call it. On that there ARPANET.
“Rust and other browser languages”.
Oh man…
I think that the key to the success of Python is that, as of today, I think it is the only language that is widely used in:
* Web applications
* Scientific computing (academia and industrial R&D)
* Big data & fintech
* DevOps
On top of that, it has wisely combined a simple syntax with the basic notions of OOP and just a pinch of functional semantics, so most people can make themselves comfortable using it.
It could have gotten an even bigger piece of the cake if it had been designed to be more embeddable (Lua seems to be championing that niche lately), but still, the fact that it can actually be compared to Bash, Java or R depending on the context and its evergrowing ecosystem (not just libraries, but tools and IDEs like IPython and so on) make a great all-rounder of it.
i have just started to learn the language.. Python commands are easily understandable.. simple. This was my first impression. That’s why it gain so much popularity i guess !!
I think it would be more interesting to see the difference between some top languages being discussed. As in for which tasks one is good while the other is not as good. What can be done in one, and can’t be done in another. Of course this should be kept in the same group, as to not compare procedural to object oriented language.
Some key parts
“Why is Python Growing So Quickly?”
Because my anaconda don’t want none….
I think it would be more interesting to compare the differences between some top languages being discussed. The stuff that matters. As in for which tasks one language is good while the other is not. Or even Syntax, which is also important. I will just write about Syntax, maybe some one can comment on the function part.
Some people dislike semicolon requirement at the end of the line. There are more reasons for it not to be there than for it to be required. So let’s call it like it is – annoyance. Spaces should not be used the way they are in Bash or Python. When it comes to syntax, when something is easy to read doesn’t always mean that what you’ve read is in fact what it is. The use of brackets, braces, quotes etc (as in C++/Java) is, once learned, makes what you see much more closer to what it is. Coupled with syntax-highlighting of any decent IDE, it makes things reliable and easy to understand. Code complexity is what makes things harder to understand. Also use of certain techniques like using conditional operator in C++ makes things look concise, but when reading pages and pages of code – inefficient for your mind to process. Looks like this:
(condition) ? (execute_if_true) : (execute_if_false) // this looks pretty, but once you put complex things inside those parentheses…
Where as “traditional” way – using if, then, else, will take up more lines but will be far better readable to a human being. Even the same human being who wrote that same code a month ago! Likewise, trying to make a computer language (Python) look like a human language might “appear” to be easier to learn, but there is no magic. You either make something happen, or use something else that YOU KNOW, will make it happen. You are not saving time here by skipping curly braces and such. Especially when syntax-highlighted these things make the code even more readable.
All of these points are pretty matter-of-taste syntax details. For syntax, though it’s certainly good to find a balance between APL-conciseness and Cobol-verbosity, it’s most of all just a mattern of what style one is used to, what they find most readable. Different syntaxes just lay out the same code in another manner – that’s never a very big difference unless you go to the extremes.
What’s much more relevant for a discussion about a language’s merits is what abstraction capabilities it offers, because only that allows you to actually have less code in the first place.
Insightful post! As someone that has worked with R and Python/Pandas for data science, the above analysis bears out what I have anecdotally seen as a shift in data science towards Python from R. One factor could be the growing popularity of deep learning, and the existence of libraries in python (e.g. Tensorflow and Theano) for implementing it, compared to R.
Theano? So 3 mos ago.
After you learn python, its bordering on insanity to use anything else. There are some very smart people who are core developers in Python and they are innovative, while the likes of C++ has put their feet up and are gloating about how they helped force OOP on the world while trying to wipe inheritance from the history books.
If you are not doing windows GUI or something that needs to be heavily optimised then you are never going to chose a C based language over python for any problem. Its a very powerful language with great libraries and is fun to write. Python is now and it is the future.
“After you learn python, its bordering on insanity to use anything else.”
I could say the same for Haskell.
To each his own.
Informational post David, as a begineer in Python & R, its any eye opener to know there is lots of growth; meaning more information is available for learning(even if some is repetitive or not useful) yet many are contributing to improve their understanding about it should be saluted. Personally I can look at a clear path to move forward & grow.
I am in a data-science-adjacent field in the manufacturing industry. I came to stack overflow today looking for help on a python project.
Data science is growing hugely in the manufacturing industry. We want to know how our products are being used, when and why they are failing, etc. We make high-value items, and we have constant electronic health monitoring of them, so we can find and diagnose failures. We also have some data coming from the manufacturing shop, such as the conditions of the various tools.
Thanks for the insights shared Moink. I’m looking for insights into machine learning applicable areas for health monitoring and diagnosis of failure of equipment. Particularly, in the domain of machine’s predictive maintenance. I’ve a few questions I was wondering if you would take time to look at. Thanks
what about swift (iOS) ? can’t see in graph
Since I am working on a Django site with data science content, I am surprised to see no connection between those two domains in your graphs. But since a site could conceivably be built with Ruby while the data science is done with R, it is conceivable that those two domains don’t have much in common even if they are two shops in the same house. Take Instagram, for example. It is my understanding this is a Django site, and clearly they are crunching a lot of data. If we assume that work is also done in Python, it does not follow that the web devs are posting data science questions and vice versa, even if the results of those two specialties are constantly influencing each other, the one being a source of data input and the other reflecting changes based on the data results.
So my takeaway is that even if Python is growing, it is widely diverse, and just like you wouldn’t go to an allergist for a broken foot, you have to specify what kind of Pythonista you are, or are looking to hire!
SO calculates these results based on the tags visited by individual user. Even though there is very likely that many organizations may use Python for data-science and web development, but the same person working on the data science and web development would be rare. I think based on that it makes sense why those two are not connected.
maybe nice for situation where software runs behind something (ea webserver) or where your doing opensource. For the rest i see little in comercialization of python projects. (dont get me wrong i like python). But maybe what these graphs also say is that most people who can program rely less on stack exchange
Its like this: Python has at some point in the past decade achieved critical mass whereby a person new to programming could become just as employable as a Python programmer as someone from any of the other traditional mainstream languages.
Once that became the case, why would someone, with nothing more than the general ambition of being a professional programmer, voluntarily choose one of the inferior languages (PHP, Java, Javascript, C#, etc) over Python, a properly designed language for use by actual Humans?
This is why I chose to learn Python a few years ago (the only palatable alternative was Ruby, but that language is clearly on the wane). I am now a upwardly mobile Django web developer, and Python continues to rightfully increase in popularity, slowly leaving the established Sacred Cow languages in its wake.
I think I made a good choice…
I think Its because python is multipurpose. Since It can be used for almost anything. at anytime. To everyday tasks in automating texts, to automating emails, even your own home. for writing apps on data science and even your simple to complex AI. For writing down apk’s for making games. for teacher children how to code. to teaching your grandma how to write down a simple buffer overflow exploit. or to simple print out hello world. to webscraping to sqldatabase automation. to network automation. to the laziest way of shutting down your laptop. python has the answer to most of your everyday needs. I hope One day ppl can write down a program in python that could resurrect the dead. lol
Organization with security data ,sensitive data will not over to community language.I’ve worked in organization like this and they had choose ADF framework- when I have asked why- at these time it was unknown framework -They answered -this framework has home .The question was .NET or Java .Serious companies will choose language or framework with support -and not community -if there is a bug the answer will -sorry:) we are not responsible-it’s community package 🙂 .Java as product of Oracle has support and If there some incident you know who is responsible or will help you.
I’am working for quite a while in a middle big software company. We once had a problem with a oracle database and it was very very difficult to find a specialist, and money was not the point. I ll experience just the opposite, we use more and more open source software, either we can solve problems our self or find some other kind of support in the community. To rely on the big companies can be very risky.
Python is a great tool for doing things fast and hence is good for scripts and preferred by researchers ( which is a good thing) but I think just like the Fotran and C algos they have produced in the past production code will be in a different language. One thing that surprises me is why python has not replaced most build / command / shell scripts we were using it for this around 2000 and I would have thought it would be more pervasive . In terms of R vs Python I see big data in R , Neural networks in Python.
I wished that I am now am expert in python
I would have written articles on my blog wisdomx13.com on python tutorials
I use both R and Python but I prefer R for analysis. One working theory I have is that the people who prefer Python come from a software engineering/programming background and the people who prefer R come from a statistics/research background. I used to work in Perl and I find R is very similar in it’s emphasis on ease of workflow and less on language strictness and conformity. For large projects with many developers, Python is great because it ensures some degree of coding style consistency. For the data science work I do (mainly analysis and publishing research), I personally find Python to just have too much overhead. But again, it depends on what you mean by “data science” (this is a big term). If you are working on the data engineering end or doing a lot with ML frameworks, Python is a more intuitive choice. Why jump through a layer of abstraction (aka an R lib that interfaces with the Python lib) when you can just work with the Python lib directly? I think this trend is indicative of a misunderstanding of what “data science” is. Many people I work with on the software engineering side assume “data science” means big data, ETL pipelines, ML/DL, etc. However there’s another side to data science people forget about, one that is more research focused, relying on “small data” (ie, statistics) and is hypotheis/scientific-method driven (not data-driven). Having used both R and Python, my impression as that R does an excellent job of catering to the end of the data science spectrum where most of the “computation” is happening in your brain 😉
Since I am learning Python about Data Science, this article gave me a great insight also a determination to maintain the path I have taken. Thank you very much.
Nice article with supporting documentation. Though, over the decades of computer science, we find new territories of exploration and the latest ‘new’ language. As I look at today’s machine learning, and its strong association with Python, I cannot wondering that as we base future decisions on past data, if we are not creating self fulfilling feedback loops and narrowing our choices and awareness of the incredible complexity and dynamically changing of the whole, systems view of the world.
Nice approach to the question. I wouldn’t have thought to use stack overflow as a proxy. As an R user, I can’t wait to read your next article as to why you chose R over Python.
I believe R is super easy to setup, unlike Python, and combined with RStudio, the best interface for data analysis. I believe if you think like a computer science major, or if want to control ever aspect of what you are doing, then Python is the way to go. Otherwise, R solves many analysis task with relatively little code. The perfect example is ggplot2…I can create beautiful graphs with 2 or 3 lines of code…I can astound with 6 or 7 lines of code.
How do you generate these nice looking graphs that so much look like FiveThirtyEight ? Could you share this code, probably it’s Python code? 🙂
Hi @ivbeg:disqus, I think the graphs for this post are generated in R, but if you are interested in generating FTE-like graphs using Python, you should take a look at this tutorial I wrote a couple of weeks ago: https://www.dataquest.io/blog/making-538-plots/
With what package was the Network of Correlated Tags produced?
N=1, but I have a data point for why pandas/python is so “popular”; It’s because pandas is completely opaque and full of gotchas. I’ve visited _hundreds_ of pandas topics in the past week or two, and every one of those visits is because some dead simple thing in pandas requires you to navigate around myriad shipwrecks. Count of times something has appeared before now? .eq, .groupby, .cumsum, .groupby, .shift, .fillna, downcast… Got NaNs in a groupby column? Make another column named “foohasNan”, leave out the the “foo” column of interest, and go to town, because pandas helpfully drops your NaN rows for you, despite data loss. I could go on, but I risk turning this into a venting session about the awfulness of pandas, when really, all I want to do is to point out that there could be an obvious factor inflating the python/pandas counts: pandas has a truly, monumentally, shamefully bad API, and the only way to understand it is to cross the minefield and get blown up a hundred or a thousand times.
I read another good article on the same topic.
https://medium.com/@dhiraj8899/top-9-reasons-why-python-may-become-more-popular-than-java-caeb316aad83