architecture,erlang,docker,elixir,microservices , Where does Elixir/erlang fit into the microservices approach? [closed]


Where does Elixir/erlang fit into the microservices approach? [closed]

Question:

Tag: architecture,erlang,docker,elixir,microservices

Lately I've been doing some experiments with docker compose in order to deploy multiple collaborating microservices. I can see the many benefits that microservices provide, and now that there is a good toolset for managing them, I think that it's not extremely hard to jump into the microservices wagon.

But, I have been experimenting with Elixir too, and I am quite fond of the benefits that provides by itself. Given that it encourages packing your code into multiple decoupled applications, and supports hot code upgrades, how would you mix docker with elixir (or erlang, for that matter)?

For example, if I want to use docker because it provides dev-prod parity, how does elixir fit in that? Given that docker containers are immutable, I lose the ability to do hot-code upgrades, right? What about blue/green deployments or canary releases?

I mean, I could just write microservices with Elixir and use them as if they were written in any other language, polyglotism is one of the benefits of microservices anyway, but then I'm not getting the full benefits of using the OTP platform, I guess that pure collaborative erlang applications are way more optimal that using intermediate queues to communicate between microservices written in different (or not) languages.

Thanks in advance.


Answer:

This is a very open question but I will try to illustrate why Elixir/Erlang may be the best platform out there for developing distributed systems (regardless if you are working with microservices).

First, let's start with some background. The Erlang VM and its standard library were designed upfront for building distributed systems and this really shows up. As far as I know, it is the only runtime and VM used widely in production designed upfront for this use case.

Applications

For example, you have already hinted at "applications". In Erlang/Elixir, code is packaged inside applications which:

  1. are started and stopped as unit. Starting and stopping your system is a matter of starting all applications in it

  2. provide a unified directory structure and configuration API (which is not XML!). If you have already worked with and configured an OTP application, you know how to work with any other one

  3. contains your application supervision tree, with all processes (by process I mean "VM processes" which are lightweight threads of computation) and their state

The impact of this design is huge. It means that Elixir developers, when writing applications have a more explicit approach to:

  1. how their code is started and stopped

  2. what are the processes that make part of an application and therefore what is the application state

  3. how those process will react and be affected in case of crashes or when something goes wrong

Not only that, the tooling around this abstraction is great. If you have Elixir installed, open up "iex" and type: :observer.start(). Besides showing information and graphs about your live system, you can kill random processes, see their memory usage, state and more. Here is an example of running this in a Phoenix application:

Observer running with a Phoenix application

The difference here is that Applications and Processes give you an abstraction to reason about your code in production. Many languages provides packages, objects and modules mostly for code organization with no reflection on the runtime system. If you have a class attribute or a singleton object: how can you reason about the entities that can manipulate it? If you have a memory leak or a bottleneck, how can you find the entity responsible for it?

If you ask anyone running a distributed system, that's the kind of insight that they want, and with Erlang/Elixir you have that as the building block.

Communication

All of this is just the beginning really. When building a distributed system, you need to choose a communication protocol and the data serializer. A lot of people choose HTTP and JSON which, when you think about it, is a very verbose and expensive combination for performing what is really RPC calls.

With Erlang/Elixir, you already have a communication protocol and a serialization mechanism out of the box. If you want to have two machines communicating with each other, you only need to give them names, ensure they have the same secret, and you are done.

Jamie talked about this at Erlang Factory 2015 and how they were able to leverage this to build a game platform: https://www.youtube.com/watch?v=_i6n-eWiVn4

If you want to use HTTP and JSON, that is fine too and libraries like Plug and frameworks like Phoenix will guarantee you are productive here too.

Microservices

So far I haven't talked about microservices. That's because, up to this point, they don't really matter. You are already designing your system and nodes around very tiny processes that are isolated. Call them nanoservices if you'd like to!

Not only that, they are also packaged into applications, which group them as entities that can be started and stopped as unit. If you have applications A, B and C, and then you want to deploy them as [A, B] + [C] or [A] + [B] + [C], you will have very little trouble in doing so due to their inherent design. Or, even better, if you want to avoid adding the complexity of microservices deployments into your system upfront, you can just deploy them altogether in the same node.

And, at the end of the day, if you are running all of this using the Erlang Distributed Protocol, you can run them in different nodes and they will be able to reach other as long as you refer to them by {:[email protected], :name} instead of :name.

I could go further but I hope I have convinced you at this point. :)


Related:


erlang processes and message passing architecture


concurrency,process,erlang,messages
The task I have in hand is to read the lines of large file, process them, and return ordered results. My algorithm is: start with master process that will evaluate the workload (written in the first line of the file) spawn worker processes: each worker will read part of the...

How does ETL (database to database) fit into SOA?


database,architecture,soa,etl,decoupling
Lets imagine, that our application needs ETL (extract, transform, load) data from relation database to another relation database. Most simple (and most performance, IMHO) way is to make link between databases and write simple stored procedure. In this case we use minimal technologies and components, all features are "out of...

simple model when requesting collection and extended model when requesting resource - how


rest,architecture,restful-architecture
I have the following URI: /articles/:id, where article is a resource on web-service and have associated model/class. Now I need to return only partial data for each resource (to save bandwidth and make for speed) when collection is requested, but when a single item is requested from collection I need...

Segregating the read-only and read-write in Spring/J2EE Apps


mysql,jpa,design-patterns,architecture,spring-data
We using Spring, Spring-Data and JPA in our project. For production servers, we would like to setup database cluster such that all read queries are directed to one server and all write queries are directed to another server. This obviously will require some changes in the way the DAOs are...

Micro Service cross service dependencies


architecture,microservices
Just to simplify my situation I currently have 3 micro services. Authentication Locations Inventory The authentication service authenticates the user and sends back a JWT access token and I use that across the other services. Its stateless and all works well. I setup locations among some other things in the...

Erlang pass-by-reference nuances


erlang,pass-by-reference
9> A = lists:seq(1,10). [1,2,3,4,5,6,7,8,9,10] 13> Fn = fun (L) -> [0|L] end. #Fun<erl_eval.6.90072148> 14> Fn(A). [0,1,2,3,4,5,6,7,8,9,10] 15> A. [1,2,3,4,5,6,7,8,9,10] If erlang internally passes by reference (see this), why does the value of A not reflect the change? What fundamental am I missing about passing-by-reference or erlang?...

WCF service architecture query


asp.net,architecture,wcfserviceclient
I have an application that consists of a web application, and mutliple windows services, only one windows service is installed depending on what version of the backend sofware is used. Currently, Data is saved by the web app in a database, then the relevant service is installed and this picks...

What's the recommended way to load an object graph from Data Access Layer?


c#,architecture,domain-driven-design,data-access-layer,software-design
From a relatively old software architecture book: In other situations, the same conceptual Get-Orders query might generate different data—for example, a collection of Order objects plus order items information. What should you do? Should you expose two slightly different GetOrders methods out of your DAL? And what if, depending on...

Libgdx: Objects creating other objects


java,android,architecture,libgdx
I have kind of a general question about a simple game architecture. In my game I have these classes: Main class who's responsible for drawing and rendering. Ball object which has a few attributes and update() function that does certain things. The main class has an array of all exist...

how to handle low_entropy exception of crypto:strong_rand_bytes(N)?


openssl,erlang
I want to generate cryptographically strong pseudorandom numbers in erlang for session IDs. There is crypto:strong_rand_bytes(N). What if it throws the low_entropy exception? From http://www.erlang.org/doc/man/crypto.html#strong_rand_bytes-1 strong_rand_bytes(N) -> binary() Types: N = integer() Generates N bytes randomly uniform 0..255, and returns the result in a binary. Uses a cryptographically secure prng...

Value from binding in LFE interpreter using Erlang


erlang,lisp,lfe
I'd like to use Lisp Flavored Erlang as a scripting extension language for an Erlang application. If you want, in a similar way GNU Emacs is configured and extended via Emacs Lisp. I know that the argument is wide and structured; but in the specific case of this question I'd...

Java generic class that contains an instance of implementation of generic interface


java,generics,interface,architecture
I am developing an independent, self contained component, that needs domain specific parts to function properly. The part of the idea is to create a generic interface, that will settle the usage of interface's implementation in another part of this component (in my example in class B). I have written...

Erlang driver erl_errno issue


erlang,erlang-driver
I am playing with erl_driver. Start callback of my driver is below: ErlDrvData drv_start(ErlDrvPort port, char* command) { char* file_name = command + sizeof(drv_name); GenTtyData* port_data = (GenTtyData*)driver_alloc(sizeof(GenTtyData)); erl_errno = gt_open(file_name, &port_data->file); if (erl_errno != 0) { // Assertion there is just to show you my intention assert(erl_errno == ENOENT);...

Fetching and updating data in mnesia


erlang,mnesia
I have multiple mnesia tuples like (GroupID is the primary key) {GroupID, GroupName, GroupType, GroupDescription, GroupTag, CreatorID, AdminID, MemberList, Counter}. MemberList = "[email protected],[email protected],[email protected]". GroupName = "Any String Value". % e.g.: "basketball" GroupTag = "comma separated values". % e.g.: "Sports,Cricket,Fifa,Ronaldo" I will pass a character or word to a function. This...

passing arrays to a function in erlang


arrays,function,erlang,arguments
I am new to erlang programming and as a matter of fact no very experienced programmer. I need to pass an array to a function and then manipulate it.. however I have so far not be this what I found. module(easy). export([myfunction/1]). myfunction([myarray]) -> %% mycode.. hier I will need...

Elixir exrm release crashes on eredis start_link


erlang,elixir,exrm
I'm fairly new to Elixir and this is the first app that I'm attempting to release using exrm. My app interacts with a Redis database for consuming jobs from a queue (using exq), and also stores results of processed jobs in Redis using eredis. My app works perfectly when I...

Where to format collections / objects


javascript,backbone.js,architecture,project-structure
From front end architectural point of view, what is the most common way to store scripts that perform transformations on collections of objects/models? In what folder would you store it, and what would you name the file / function? Currently I have models, views, controllers, repositories, presenters, components and services....

How to invoke Erlang function with variable?


erlang
4> abs(1). 1 5> X = abs. abs 6> X(1). ** exception error: bad function abs 7> erlang:X(1). 1 8> Is there any particular reason why I have to use the module name when I invoke a function with a variable? This isn't going to work for me because, well,...

Erlang syntax error unclear


function,variables,if-statement,functional-programming,erlang
I just got started with Erlang. I am trying if statement. I found out one particular behavior which I do not understand. the following statement does work perfectly. some_comp(Arg1) -> if (cal(Arg1)>50000)->'reached'; true -> 'invalid' end. cal(Arg2)-> %% some calculation. However the following shows an error syntax near if: some_comp(Arg1)...

What is the difference between .app and .app.src files in Erlang?


erlang
I found these description in the Erlang doc and learnyousomeerlang.com, but not sure which one is the "Application Resource File". When should I use .app and .app.src? 7.3 Application Resource File http://www.erlang.org/doc/design_principles/applications.html The Application Resource File http://learnyousomeerlang.com/building-otp-applications...

C# Code design / Seperate classes for each TabControl


c#,oop,architecture,software-design,code-design
My main problem is that my tool grows and grows and I start loosing the focus on the different parts of my code. The main-Form got a docked tabControl at fullsize. I got 5 different tabs with for really different functions. So I can say my tool is splitted into...

How to model data for in-memory processing


java,performance,design,architecture,data-modeling
I have a lot of static data (i.e. read only data, which is not transactional) which gets updated only once in few days. I have to support searches on that data (api calls, not sql). So I am thinking I will just load it in Memory, and refresh the in-memory...

How to asynchronously send data to a client via a different application path?


web-applications,architecture,scalability
I am currently researching large scale application optimisation and scaling, and through my research I have gotten to grips with the standard ways of doing large scale with DNS Round Robin for splitting the load across load balancers, using load balancing to divide traffic across web-servers like Nginx, which again...

ctrl+G in erl doesn't work


unicode,encoding,utf-8,erlang,docker
I'm trying to interconnect erlang nodes, but entering ctrl+G doesn not work: Eshell V6.4.1 (abort with ^G) 1> ^G Eshell V6.4.1 (abort with ^G) 1> ^G Eshell V6.4.1 (abort with ^G) 1> ^G Eshell V6.4.1 (abort with ^G) any idea why this can happen? I was thinking about locale settings,...

Architecture for creating a JavaScript framework


javascript,architecture,frameworks
Around one year ago we started a web system that over the time has grown quite a bit. From the beginning the goal was to build reusable code that would speed up the development of future projects, and it has. With every new project, the reusable code from the previous...

Getting a sibling process in Elixir


erlang,elixir,otp
I have an Elixir/Erlang process tree: parent (Supervisor) ├── child1 (GenServer) └── child2 (GenServer) child1 (a DB client) has information that child2 needs to use. What's a good way to pass a reference from the Supervisor process to child2 so that child2 will always have a valid reference to child1?...

Should I create another model for admins? Or what's the best way to do it in Ruby on Rails?


ruby-on-rails,database-design,architecture
I'm developing an app, which requires two types of users regular user and admin. Regular users have roles in the app, and an admin doesn't have any roles, he just should have an access to an admin panel. What's the best way to implement this task? ...

os kern error : “ld: symbol(s) not found for architecture x86_64”


c++,architecture,compiler-errors,x86-64,clang++
I have looked all over Stack Overflow and other websites about this famous error, and all of them are very specific, and in my case I cannot find a solution. I am making an ncurses application and when i try to compile it, it causes the following error: Undefined symbols...

What does BEAM stand for in iex for the Elixir programming language?


architecture,elixir,acronym,beam,lightweight-processes
I'm sort of curious as to what the B. E. A. and M. stand for. I recall seeing an explanation of the acronym BEAM, but I have not managed to find it again. It comes up in error codes: ➜ gentoo iex Erlang/OTP 17 [erts-6.4.1] [source] [64-bit] [smp:8:8] [async-threads:10] [kernel-poll:false]...

erlang os:cmd() command with UTF8 binary


unicode,encoding,erlang,utf
I'm trying to get an Erlang function to execute a bash command containing unicode characters. For example, I want to execute the equivalent of: touch /home/jani/ჟანიweł I put that command in variable D, for example: io:fwrite("~ts", [list_to_binary(D)]). touch /home/jani/ჟანიwełok but after I execute: os:cmd(D) I get file called á??á??á??á??weÅ?. How...

Subtypes of an interface only compatible with a subtype of another interface


oop,design-patterns,design,architecture,solid-principles
I've been scratching my head for a long time on a problem and I still don't know what would be the best solution. Since the application domain is very technical, I'll illustrate my problem with a simple example. Suppose I have the following interfaces: public interface Animal { public void...

Erlang spawning large amounts of C processes


c,multithreading,lua,erlang,ffi
I've been looking into how I could embed languages (let's use Lua as an example) in Erlang. This of course isn't a new idea and there are many libraries out there that can do this. However I was wondering if it was possible to start a Genserver with state which...

Why doesn't spawn link cause the calling process to die?


erlang,erlang-shell
From the example given here, Erlang and process_flag(trap_exit, true) -module(play). -compile(export_all). start() -> process_flag(trap_exit, true), spawn_link(?MODULE, inverse, [***0***]), loop(). loop() -> receive Msg -> io:format("~p~n", [Msg]) end, loop(). inverse(N) -> 1/N. If I run it as, A = spawn(play, start, []). The spawned process <0.40.0> dies as it is suppose...

Difference between switch & bus architecture?


architecture,operating-system
I was just going through a operating systems textbook but I got stuck at switch architecture . Please explain what it is and how it is different from bus architecture ?...

Data flow of MVC application architecture


c#,asp.net-mvc,architecture
Attempting to validate the approach for data flow in an MVC application that i am cleaning up and streamlining, (after a bit of refactoring) things currently looks like the diagram below (Data flow indicated by arrows). and some parts are written to access the the repository services jumping over layers....

Does Erlang has Map?


erlang
It seems that Erlang introduced maps in version R17A. But, if I go to downloads I only see version 17.5, there's no version R17A. So, is it released and stable yet? Does the latest stable Erlang has Map support?...

Client-Server architecture: 100% Android (Android as a server) or J2EE+Android?


android,java-ee,architecture,client-server,distributed-computing
Context I am considering going into a client-server architecture with Java. The idea is that several Android tablets (let's say around 15) need to display a content from a server. Content can vary times to times (e.g. day display v/s night display). Furthermore, tablets will also display a Yes /...

Some doubts related this Swing MVC implementation. Opening a database connection should be a Controller task?


java,swing,design-patterns,model-view-controller,architecture
I am following a Java tutorial related to the implementation of the observer pattern (using Swing) and I have some doubts. My doubts are not related to the observer pattern but about the architecture of this tutorial application (that is based on something like an MVC logic) So it contains...

How to list all the bucket types in riak?


erlang,riak
Using Riak Client(erlang), I can list all the buckets of the default bucket type. But how can I list all the bucket types? Or If I can't do this by the client, how can I find out about this?

MVC Web application architectural concern


c#,asp.net-mvc,web-applications,architecture
I have inherited an MVC application designed along the above project structure. the application uses the Unity framework for dependency injection, and the user interactions go upstream to the database in the following order View -> Controller -> ViewModels -> Repository Services -> ORM -> Database The infrastructure components...

Riak mapReduce fails with > 15 records


mapreduce,erlang,riak
Problem I've been learning Riak and ran into an issue with mapReduce. My mapReduce functions work fine when there's 15 records, but after that, it throws a stack trace error. I'm new to Riak and Erlang, so I'm unsure whether it's my code or it's Riak. Any advice on how...

Mnesia pagination with fragmented table


pagination,erlang,mnesia
I have a mnesia table configured as follow: -record(space, {id, t, s, a, l}). mnesia:create_table(space, [ {disc_only_copies, nodes()}, {frag_properties, [ {n_fragments, 400}, {n_disc_copies, 1}]}, {attributes, record_info(fields, space)}]), I have at least 4 million records for test purposes on this table. I have implemented something like this Pagination search in Erlang...

Getting SSL related error against my request to Ejabberd


android,sockets,ssl,erlang,ejabberd
Following is the code snippet where i open a socket to write APNS notifications on: get_socket()-> %%Options Options = [{certfile, ?Cert}, {keyfile, ?Key}, {mode, binary}], %%ssl connection ssl:connect(?Address, ?Port, Options, infinity) . close_socket(Socket)-> ssl:close(Socket). I am getting the following crash in my ejabberd.log file 2015-06-05 12:33:17.112 [error] <0.3134.0> gen_fsm <0.3134.0>...

Type errors in dialyzer in function with specified type


erlang,dialyzer
I am receiving an error in dialyzer when it analyzes the following function. -spec do_request(Method, Type, Url, Expect, Headers, Body, Client) -> Response::client_response() when Method :: method(), Type :: content_type(), Url :: url(), Expect :: status_codes(), Headers :: headers(), Body :: body(), Client :: #client{}. do_request(Method, Type, Url, Expect, Headers,...

Why doesn't JavaScript get its own thread in common browsers?


javascript,browser,architecture,software-design
Not enough that JavaScript isn't multithreaded, apparently JavaScript doesn't even get its own but shares a thread with a load of other stuff. Even in most modern browsers JavaScript is typically in the same queue as painting, updating styles, and handling user actions. Why is that? From my experience an...

Fast mutable objects in Erlang


erlang
What is the best approach to store and manage high-performance mutable objects in Erlang? Assume that I want to write really simple online game server with realtime gameplay. Somehow I need to represent player's state in Erlang memory. For example, it could be just a simple tuple like {name, "Bob",...

erlang message passing architecture


process,erlang,messaging
In erlang if two processes A and B are sending message to a process C simultaneously. Will there be a race condition? C ! {very large message} sent by A C ! {very large message} sent by B Will C receive the complete message from A and then proceed for...

First word of binary string erlang


functional-programming,erlang,pattern-matching
I need to do something like this <<"Some text in binary">>, and it should return <<"Some">. How can i do it without split function, only by pattern matching and select/if cases with Erlang. Thank you.

Does the pattern [_|_] in Erlang mean anything specific?


erlang
I am learning Erlang from Learn You Some Erlang and I've seen the pattern [_|_] twice already but couldn't find any info on it. This usage seems superfluous because omitting it from (1) and subsituting it with _ in (2) yields the same result without degrading readability. It is my...

Erlang: Returning a function from a function


erlang
I know Erlang supports anonymous functions. My question is, can I return a function from a function then call that returned function from outside? If so, how do I do it? I know this is possible in many languages such as C and Python. Here is what I tried to...