Motoko, a programming language for building directly on the internet
At Dfinity, we’re building the Internet Computer, a decentralized cloud computing platform that we conceive as a seamless software universe in which developers can deploy applications and services directly on the Internet. To realize this vision, we decided on WebAssembly as the lingua franca of the platform’s execution environment, so that developers can program it in any language that compiles to WebAssembly.
To offer a seamless developer experience, we also felt it was important to create a specialized programming language, called Motoko, that is designed to directly support the programming model of the Internet Computer, making it easier to efficiently build applications and take advantage of some of the more unusual features of this platform.
We’re excited to share with you a little about what makes this project so special.
First, we have to talk briefly about WebAssembly—a.k.a. Wasm (yes, correctly spelled without all caps). As you may be aware, Wasm is a newish low-level code format that aims to be portable, safe, and efficient. Its initial use case has been the web, but the name actually is a misnomer: when we designed Wasm in the W3C Working Group, we carefully did it as an open standard and a universal platform. That is, it is not aimed at any specific programming language, paradigm, computing environment, or platform, and we made sure that it is not at all tied to the web. So it is absolutely no accident that Wasm is seeing adoption in many other environments, such as cloud computing, edge computing, mobile, embedded systems, IoT, and blockchains.
There were many, many design considerations that went into Wasm, some obvious and some rather subtle. Too many to go into here. A fairly comprehensive discussion of Wasm’s technical goals, design choices, formal semantics, and implementation techniques can be found in a scientific article that we published in Communications of the ACM (an older and more technical version of this article is freely accessible here).
Wasm’s main difference compared to other virtual machines is that it is not optimized for any specific programming language but merely abstracts the underlying hardware, with a byte code directly corresponding to the instructions and memory model of modern CPUs. On top of that, Wasm supports sandboxing through strong modularity and a rigid mathematical specification that ensures that execution is safe, free of undefined behaviour, and (almost) entirely deterministic. Moreover, these properties actually have a machine-verified mathematical proof!
Altogether, these properties were intended to make Wasm an attractive choice for a wide range of environments and use cases that have high expectations for portability, safety, generality, and performance—such as Dfinity’s Internet Computer.
Creating the Internet Computer
The Internet Computer is a decentralized cloud computing platform that will host secure software and a new breed of open internet services. It uses a strong cryptographic consensus protocol to safely replicate computations over a peer-to-peer network of (potentially untrusted) compute nodes, possibly overlayed with many virtual subnetworks (sometimes called shards). Wasm’s advantageous properties made it an obvious choice for representing programs running on this platform. We also liked the idea of not limiting developers to just one dedicated platform language, but making it potentially open to “all of ’em.”
That’s the theory, anyway. In practice, porting an existing programming language to Wasm is not entirely trivial. Obviously, it requires implementing a new compiler backend. That’s fun, but the effort doesn’t end there: it also requires porting the language’s runtime system and library primitives. And there are still a few features, especially ones relevant to more high-level languages, that cannot currently be implemented in Wasm easily—for example: threads, coroutines, exceptions, and tail calls. While various proposals to enrich Wasm with respective functionality are on the horizon, they have not yet been finalized for standardization.
Although there are many experimental language implementations targeting Wasm already, most are not yet ready for prime time. The ones that are include low-level systems languages like C/C++ and Rust. These are certainly great for their use cases, but they are less-than-ideal tools for developing high-level applications for the Internet Computer, where accessibility, productivity, and high assurance tend to be more desirable than manual meddling with memory management.
On some platforms, including the Internet Computer, there are additional hurdles that need to be overcome to run Wasm, and they have to do with the limitations of the computing environments they provide. For example, Dfinity’s Internet Computer has very little similarity to a conventional operating system: there is no functionality like files, I/O, or other capabilities often taken for granted in language implementations and liberally used in runtimes or libraries. That means that porting an existing language is more than just a question of tweaking code: you may need to find new means to replace uses of missing platform functionality, remove them, or make different design choices altogether. Efforts like WASI try to address this problem to some extent, but are still in their infancy.
Unavoidably, these factors make a language port to Dfinity’s Internet Computer substantial work, even when adopting a language for which a generic Wasm port already exists.
At the same time, a language for the Internet Computer needs to provide access to the platform’s main concepts: a distributed programming model with asynchronous message passing, notions of resources like cycles (a.k.a. gas), and a few other idiosyncrasies. Sure, they could all be made available as libraries, but a language that natively includes appropriate constructs can deliver a much more seamless programming experience.
So if we have to do work anyway to get off the ground, why not apply ourselves to creating something that can deliver an optimal user experience and convey our vision for how to program the Internet Computer?
That is why—despite all the risks of creating yet another language—we decided to create Motoko. We wanted a language that is safe, easy to use, and seamlessly exposes the concepts of the platform, as well as one that looks sufficiently friendly and accessible to most programmers. Currently, that latter goal makes it practically inevitable that it’s firmly in the semicolons-and-curly-braces camp of languages. And no suitable language existed in this camp.
But Motoko’s rather conventional skin is only superficial: its interior is that of a modern language. For example, every construct is an expression, it has closures, it has variant types and statically checked pattern matching, it has garbage collection, and of course it has a flexible type system that is actually sound, i.e., it really guarantees the absence of certain errors like crashes, undefined behaviour, misinterpreting data, or simply missing a case in a switch. No holes!
At the same time, we intentionally tried not to be fancy or reinvent the wheel, but rather built on a wealth of history, both practical and theoretical, and acknowledged the lessons that have been learned over decades in this field. Besides putting together a coherent mix of well-understood features, Motoko’s design incorporates many small decisions to minimize foot guns and err on the side of safety, e.g., numbers cannot overflow by default, locals are immutable by default, concurrent execution is atomic by default, null cannot occur by default, fields are private by default, and so on. Oh, and there is no inheritance, only subtyping.
Implementing these parts of Motoko and compiling them to Wasm is conventional compiler craft. The Motoko compiler, written in OCaml, uses a typed intermediate representation, a few transformation passes, and spits out Wasm byte code. The generated Wasm module includes a small runtime system, written in C and Rust, that mainly implements a simple garbage collector using the Wasm memory as its heap. That wasn’t hard, but surely there is much potential for improvement here.
The central feature of Motoko, however, is its direct support for actors, in both syntax and type system. The actor model is a well-known concept that is 40+ years old, but sadly, it has barely made it into mainstream languages. An actor is like an object (and in Motoko, even looks like one), in that it encapsulates private state along with a set of methods to process messages that can be sent to it. But all message sends are asynchronous. Consequently, unlike conventional methods in OO, actor methods do not have results. Moreover, all messages are received sequentially by an actor—that is, it has an implicit message queue and methods execute atomically, even when messages are sent concurrently.
Actors are a great model for concurrent programming because they automatically prevent race conditions (thanks to atomicity and encapsulated state) and deadlocks (because execution never blocks), and hence rule out many concurrency bugs. All that without requiring programmers to ever define a lock. Actors are also a great model for distributed programming, because asynchrony naturally deals with the latency involved with sending a message to a potentially remote receiver. And finally, actors are a great fit for Dfinity’s Internet Computer, where applications are deployed in the form of so-called canisters—essentially, actors represented as Wasm modules that can communicate across subnetworks. It turns out that Wasm’s module concept is a nice fit for this because we can directly interpret module exports as actor methods. So a Motoko actor compiles to a Wasm module, where the methods become exported Wasm functions with special parameter conventions defined by the platform.
In short, an application in Motoko is an actor (or several), which in turn is a big asynchronous object compiled into a Wasm module. With Wasm’s notion of memory, such an actor can immediately manage up to 4 GiB of internal state, although this can be enlarged further by linking multiple Wasm modules that each have their own memory. We are curious to see how quickly the first users will run into this memory limit.
To make asynchronous programming more convenient and allow expressing it in sequential “direct style,” Motoko adopts another 40+-year-old idea from the annals of programming language research, though one that fortunately became a bit more popular recently: futures (also called promises in some communities). In Motoko, they materialize in the form of “async values,” values of type `async<T>` that are produced by expressions prefixed with the `async` keyword. In particular, a function body can be an async expression, thereby naturally replacing the more monolithic concept of an “async function” that exists in some other languages.
With that, actor methods are allowed to have results after all—as long as those are futures. Futures can be awaited to get their value, but only inside another async expression, akin to async/await monads as known from other modern languages.
The Motoko compiler implements this via a traditional CPS (continuation passing style) transformation, turning each await point into a separate Wasm function (plus some closure environment) representing the continuation of the await. In fact, it’s double-barrelled CPS, because every message can also have a failure reply with a respective failure continuation. By convention, a method with an async result is one that sends a reply message carrying the result values as arguments. This message is received by the created continuation function, which can then resume the execution it has captured. Waiting for a reply doesn’t block an actor—it can freely receive other messages in the meantime.
Another important consideration for Motoko was allowing developers to utilize blockchain technology without having to learn an entirely new type of computing. So we took out all of the special knowledge that you might need on the current breed of blockchain programming languages. For example, there is no observable notion of block or block height, no explicit constructs for updating state on the blockchain, nor is there other API for writing data to persistent storage, like files or databases (although that could be emulated as a library). Instead, the Internet Computer implements orthogonal persistence—yet another old idea where a program has the illusion of running “forever” and its memory staying alive (at least until it is explicitly taken down). In Motoko, this means that developers do not have to worry about explicitly saving their data or bother with files or an external database: whatever values or data structures are stored in program variables will still be there when the next message arrives, even if that is months later.
The platform takes care of transparently saving and restoring the private state of a canister between method invocations. That was relatively easy to retrofit onto a Wasm engine, because the state of a Wasm module is clearly isolated in a module’s memory, globals, and tables. For the most part, it is sufficient to watch Wasm memories with the use of virtual memory techniques exposed by operating systems. This way, the platform knows when pages in such a memory have been modified and can take whatever measures are necessary to persist the dirty pages, as well as hashing them for the distributed consensus protocol.
Beyond Motoko: Interface definitions
Because the Internet Computer runs Wasm, Motoko is just one option for creating an application—and intentionally so. We are looking forward to making other language choices available. Even then, because each language will uniformly compile to canisters represented in Wasm, these canisters can freely communicate with each other through message sends regardless of their source language.
To make such interoperability well-defined, we have also introduced a generic interface definition language (IDL) named Candid that is independent from Motoko. It describes the set of messages understood by a canister and what type of data is sent along. Data is described in Candid by a combination of canonical data types (numbers, text, arrays, records, variants, functions, references to other canisters) that are independent from the Motoko type system or that of any other programming language.
Phew, yet another type system? Well, programmers will probably be pleased that the Motoko compiler can automatically consume and produce such interface descriptions for actor exports and imports and map them from and to corresponding Motoko types. It also automatically generates the right Wasm code to serialize and deserialize the argument data for each message, transparently interconverting Motoko’s internal representation with the binary format that Candid specifies.
This way, Motoko programs can communicate with external canisters in a typeful manner and express remote invocations as if they were local objects in the program. And that is regardless of whether the remote canisters are written in Motoko or, say, Rust; the interface description of a canister is enough as type information. Besides mere convenience, interfaces also provide a strong form of modularity, where programs can be type-checked against other actors/canisters without having access to their implementation.
Our goal is that the Internet Computer will become a multi-linguistic platform where all languages have equal rights, can interact seamlessly across canister boundaries, and Motoko is just one choice among many. This is important to make the Internet Computer platform an open one.
Wasm has so far proved to be a versatile code format to achieve this goal. We especially benefit from its simplicity, modularity, and safe and deterministic semantics. But despite these nice properties, porting compilers and libraries, let alone applications between different Wasm ecosystems, is not as straightforward as one might hope, since it involves so much more than just bare code. But Wasm is still young, and certain barriers are to be expected.
The biggest upcoming Wasm feature we are eagerly awaiting is the advent of first-class reference types and function references. That will make for a much cleaner system API, through which Wasm modules (and hence Motoko programs) talk to the Internet Computer platform). Interested programmers can find more details about the SDK here and contribute to the Motoko base library via GitHub here.
What is the etymology behind the name choice?
Probably Major Motoko from the anime Ghost In The Shell, who becomes one with the internet.
I love the GITS reference and, Motoko is my dog’s name.
In case you need brand face she’s https://m.facebook.com/motoko.ku.7 and probably the cutest shiba inu around
oh and the features of the lang seem pretty cool too 😉
Google “Ghost inthe shell”
likely inspired by the main protagonist of the popular anime. https://en.m.wikipedia.org/wiki/Motoko_Kusanagi
Good to read.
I think they named it after the character from Ghost in the Shell.
Pretty sure it’s a reference to Motoko Kusanagi from Ghost in the Shell.
Another language, which is a possible alternative to Go, Motoko, Gleam, Gluon, Nim, etc is Slider, available at http://mrsename.blogspot.com/
Is this a ghost in the Shell reference?
So, where is the car analogy? Where are the enterprise ‘hello world’ and CS-101 task list examples?
So what problem does the internet computer solve?
Well, it gets this company a bunch of VC money to buy kegs and fancy offices while complaining about throwing around words like “product/market fit” and various buzzwords
The problem of Yet Another Startup trying to drum up a few millions before making an exit.
Call me cynical, but I see too many buzzwords and too little concrete information for me to think anything else about it.
I read the high level, thorough explanation of a new programming language.
It solves the scaling problems Ethereum faces. While doing that it also replaces the traditional tech stacks with a single tech stack that is written from the ground up to provide for extensibility so that any programming language that can be compiled to WebAssembly can be built for the Internet Computer if you also define an interface so it can interact with the Internet Computer’s protocol (how to define it is well documented and provided for at Dfinity’s website and Github repos). However, this is all done for you if you use the language developed for the Internet Computer, Motoko, and the DFX SDK Dfinity has also developed and provides will build both the corresponding WebAssembly and Candid IDL interface spec for you from your Motoko smart contract code; the DFX SDK also provides functionality for developing, deploying and managing your smart contracts. Additionally, Dfinity has provided apps and code that you can clone and get up and running and using the Internet Computer: the list is –quite extensive– so I suggest you goto their website and see for yourself.
This means not only can you as a single developer do most of what JAM/LAMP/MEAN does for you but within a single layer, and this is also able to run completely on chain so that the app you build can wholly be on chain and a true decentralized app*.
What’s more, is because of its architecture other blockchains can benefit from the capabilities it provides and thus realize whatever potential they may hold. In fact this provides a more realistic approach to cross-chain interoperability because, for instance, if Cosmos IBC was ported over, which it can be since, like I said any language able to compile to WebAssembly can run on the Internet Computer, we’d be able to begin to see real progress towards this end. And honestly I’ve barely even scratched the surface.
Not only all that, but if you’d like you can run your own node, even fork your own subnet that runs completely independently: all the source code has been made publicly available. If you spend some time researching how it works you might see the potential it offers in making blockchain technology a tool any developer can really use.
Granted, it’s got its own host set of problems like any technology does; but unlike many technologies already overwhelmed by legacy inertia (even very recent ones), if you want to become a part of what’s to come you can download the sdk and become an author of the future unfolding. So if you still think this is a rugpull or a sham, you may just have personal bias for whatever reason, and I would encourage you to let go of that emotional buffer blocking you, so you can participate and take action in the development of what’s to come. While you may want to nay say or disbelieve, the reality is Dfinity has done tremendous work towards making blockchain technology not only more accessible, but even making an implementation of blockchain technology that other blockchains can benefit from.
Disclaimer: I don’t work for Dfinity. I also like other blockchains. Imagine what Cosmos IBC could do if built to leverage ICP. For that matter, realizing Cardano’s PAB may be a whole lot easier with a toolset designed to run and develop on and for blockchain tech. What a potent combination that would be.
*One note: Dfinity, like many other blockchains, has dedicated computation power in data centers around the world. While some may see this as antithetical to decentralization, how else are you going to get a scalable blockchain network up and running; in this case you can think of it as a performant CDN capable of natively handling cryptocurrency whose front-end runs natively on chain until we can do better. But that takes work and dedication which is why again, I am pointing out that this not some rugpull but a viable tool you can use to start developing blockchain technology.
Is the name based on GiTS?
Maybe Andreas watched GITS.
This is indeed best of the best 👌 this is another Satoshi kick.
How is security addressed?
Great article, just missing one thing: a getting started link. (I’d suggest https://sdk.dfinity.org/docs/quickstart/quickstart.html)
It is here: https://sdk.dfinity.org/docs/developers-guide/tutorials-intro.html
Sounds Japanese –
moto = cause, source, substance
ko = offspring
Agreed, but ko(コ) can also be a signifier that it’s a woman’s name. Very likely it’s based off the anime character.
This sounds really interesting! As I was reading I was thinking of mobile applications running on an intermittent network. Motoko seems like it would be a great choice for apps that need work done on the cloud. The part about automatic persistence is especially exciting, it’s the first time I’ve seen persistence abstracted in such a way. Sometimes I don’t really care how data is stored, I just need it persisted. For data that is well-structured and could be a huge data set, I’d still use a relational database for fast lookups and integrate that into the Motoko ecosystem using the IDL unless it’s already been done.
Awesome stuff, thank you for writing this! I’m going to go read more on the GitHub page now.
I will second the request for a little sample of code.
Sounds similar to Scala
Looks very demanding
want to dive..
Please tell me how can I know more about it and learn it?
More explanations are available here :
I hope you have infinite resources (time and money) to build a new language from scratch, along with a new platform and a new developers community.
Otherwise, the project is doomed.
First of all, I love the idea and will definitely give it a try.
Nonetheless, I have a few questions:
.. regarding the source code availability:
The code for all that (except the Motoko Base Library) not seems to be open source right now.
* Is it planned to release the code – at least under a license that allows public code reviews?
.. regarding the runtime:
> The platform takes care of transparently saving and restoring the private state of a canister between method invocations.
* Does it store the messages (“Update Calls” and/or “Query Calls”) as well or will I need to store them somewhere within the actor’s state
if I need them for further reference (e.g. for Event Sourcing)?
Also one actor compiles to one Wasm module, and such a module can handle 4 GiB of memory.
* Does it behave like an append-only store and will eventually fill up just by executing instructions, or are the old pages “swapped away”
to somewhere else so that I can always use the available 4 GiB for current program data?
* What can I do if my default of 4 GiB per actor is not enough?
.. Do I need to spin up new actors/modules and “shard” my data over all of them (and also have to provide the methods to do so)?
.. Can I just attach another module without actor and use only the allocated memory transparently from the initial module?
.. Will (at some point) the runtime handle that automatically for me?
.. regarding the Canister lifecycle:
> Instead, the Internet Computer implements orthogonal persistence–yet another old idea where a program has the illusion of running “forever”
> and its memory staying alive (at least until it is explicitly taken down).
Studying the documentation for the SDK (https://sdk.dfinity.org/docs/developers-guide/working-with-canisters.html), I couldn’t find a method to do
so (take down Canisters).
* So how can a user (developer/company) of the system comply with the regulatory requirements to permanently delete data on request of a user as
long as its not needed for any other legal reasons anymore (GDPR/CCPA)? Is a so called “logical deletion” enough for those cases?
.. regarding the gas/billing:
> Gas reflects the real costs of operations including resources such physical hardware, rack space, energy,storage devices, and bandwidth.
> In simple terms, a unit of gas represents the cost of executing a single WebAssembly instruction.
* If a Canister is available “forever”, are the costs for it’s storage (with all of its execution history) already included in the Gas-price
at the time of execution?
* Does the cost of the concensus-mechanism and message-routing adds to the Gas consumption or is it really only the computation
(with the associated storage costs for the per-op orthogonal persistence) in each data-center?
.. regarding the infrastructure:
* Do I/my customers need to trust the involved data-centers or their staff? Can the messages be encrypted on the way in- and out of my Canisters?
.. regarding the business- and governance model (for the data centers, as well as for the NNS):
From the outside, the Internet Computer looks like a single logical entity that is internaly made up from different data-centers,
each with their own operator.
The Internet Computer itself faces demand from outside and has “limited” capacity to fulfil the demand.
Depending on the inside-capacity and the outside-demand, it regularly calculates a new price for the tokens that I as a customer would need
to have in order to pay for my application’s gas fees.
* How can a healthy competition between the data-centers inside the Internet Computer continue if they act as a single organization?
I hate that headline so hard.
You can’t build, deploy or run any software directly to a conceptual thing like “the internet”.
> To realize this vision, we decided on WebAssembly as the lingua franca
> of the platform’s execution environment, so that developers can
> program it in any language that compiles to WebAssembly.
WebAssembly – the next UNCOL?
“Internet Computer” – because “The Network is the Computer” was already taken…25 years ago.
The candid link is broken.
Paragraph @Mokoko ‘holes’ link is broken.
Thanks for the heads up! I removed the link.
I have created a charitable foundation and I want to create a decentralized infrastructure around it, which will use the NTF as a tool for governance, financing, and controlling the activities of the foundation.
We need to develop two smart contracts that will interact with each other:
NFT minter SC and NFT governance SC
We are looking for JS developer, who could help us in that job