FAQ Database Discussion Community
queue,distributed,distributed-computing,distributed-system
What are architectural patterns/solutions that make distributed queues tick? Please share for both ordered and non-ordered types....
scala,apache-spark,bigdata,distributed-computing
What is the difference between the following transformations when they are executed right before writing RDD to a file? coalesce(1, shuffle = true) coalesce(1, shuffle = false) Code example: val input = sc.textFile(inputFile) val filtered = input.filter(doSomeFiltering) val mapped = filtered.map(doSomeMapping) mapped.coalesce(1, shuffle = true).saveAsTextFile(outputFile) vs mapped.coalesce(1, shuffle = false).saveAsTextFile(outputFile)...
java,rmi,distributed,distributed-computing
I'm trying to distribute a simple room booking system that I implemented, I tried researching and this is what I came up with.. but When I run them and enter different port numbers the client still runs..shouldn't it only run if the port number matches the one I entered for...
android,java-ee,architecture,client-server,distributed-computing
Context I am considering going into a client-server architecture with Java. The idea is that several Android tablets (let's say around 15) need to display a content from a server. Content can vary times to times (e.g. day display v/s night display). Furthermore, tablets will also display a Yes /...
vba,excel-vba,distributed-computing,memory-mapping
I am trying to build a distributive computing system that uses memory mapping files to coordinate work among several networked PCs all via VBA. Put another way, I want to get a group of networked computers to do work at the same time in a coordinated way on a single...
architecture,system,scalability,distributed-computing,trie
Does anyone know how I might scale a Trie across multiple machines? Say the first machine runs out of space and I need to add more words from a very large dictionary, what might I do to add more words? (I am a Java thinker, but I believe the answer...
performance,configuration,apache-spark,distributed-computing,spark-streaming
My Spark cluster has 1 master and 3 workers (on 4 separate machines, each machine with 1 core), and other settings are as in the picture below, where spark.cores.max is set to 3, and spark.executor.cores also 3 (in pic-1) But when I submit my job to Spark cluster, from the...
java,user-interface,jbutton,distributed-computing
I am writing GUI for a chat, and I have problem I can't seem to find a solution. When button Send is clicked variable OKpressed should change to true and in function getUserInput it should recognize it changed but it doesn't.. It's acting like it still says false.. I tried...
distributed,distributed-computing,experimental-design
For a master thesis in the field of distributed computing (think of hadoop and two-level schedulers like Mesos) I'm setting up various experiments on a university cluster. However I'm already piling up bash scripts which function as a driver for the experiment. I miss composability and reuse between subparts of...
hadoop,distributed-computing,cloudera,yarn,cloudera-cdh
In our YARN cluster which is 80% full, we are seeing some of the yarn nodemanager's are marked as UNHEALTHY. after digging into logs I found its because disk space is 90% full for data dir. With following error 2015-02-21 08:33:51,590 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Node hdp009.abc.com:8041 reported UNHEALTHY with details: 4/4...
distributed,distributed-computing,zookeeper,distributed-system
What techniques/tools can be used to implement a distributed system with these requirements: At a given time, the system can be in one of 3 states: SYNCING, COMPUTING, or IDLE. Each node in the system can receive two instructions: sync() and compute(). A sync() instruction will be sent to all...
java,distributed-computing,cqrs,failover,xa
Good time guys! We have a pretty straightforward application-adapter: once in 30 seconds it reads records from a database (can't write to it) of one system, converts each of these records into an internal format, performs filtering, encrichment, ..., and, finally, transforms the resulting, let's say, entities into an xml...
linux,sockets,distributed-computing
I'm writing a distributed system and I want to test it on my machine. I created several ip addresses on the interface lo using ip addr add ip_add dev lo. I have binded all servers to their specific addresses and now I want my servers to connect to each other...
matrix,apache-spark,distributed-computing
I am very new in Apache Spark Scala. Can you help me with some operations? I have two distributed matrix H and Y in Spark Scala. I want to compute the pseudo-inverse of H and then multiply H and Y. How can I do this? ...
asynchronous,distributed-computing,middleware
I'm just getting into the concept of a Distributed System and its advantages and disadvantages. In the book I'm reading it discusses the complexity of a Distributed System and that they are inherently complex, it lists the following as potential reasons for complexity; Heterogeneity Asynchronous communication Partial failures What I...
java,machine-learning,bigdata,distributed-computing
My project takes very long time at running, I made threads and distributed data and processing on my processor cores, But, still takes long time, I tried to optimize the code as i can, How can i distribute computing on multiple laptops?
akka,zeromq,distributed-computing
The Akka doc talks about a variety of seemingly inter-related Akka technologies without distinguishing much between them: Akka Networking Akka Remoting Akka Clustering The Akka ZeroMQ module My understanding is that "Akka Networking" is simply a module/lib that gives Akka the ability to speak to remote actor systems over TCP....
distributed,distributed-computing,mesos,mesosphere
I have a use case where I have 20-30 frameworks runnings on mesos cluster that has over 200 nodes. A lot of the times mesos is offering resources to frameworks that do not want any offers at all. While doing that, it is offering little resources to frameworks that actually...
distributed-computing,distributed-system,distributed-algorithm
I understand that failure detectors in asynchronous systems are basically classified as (eventually)perfect/(eventually)strong and how those classes are defined, but I kind of struggle to get the intuition behind it. Suppose I have a concrete implementation of a failure detector, which periodically listens for heartbeat messages from each process. If...
haskell,cloud,distributed-computing,cloud-haskell
The Problem Hello! I'm writing in Cloud Haskell a simple Server - Worker program. The problem is, that when I try to create ManagedProcess, after the server disovery step, my example hangs forever even while using callTimeout (which should break after 100 ms). The code is very simple, but I...
database,distributed,distributed-computing,data-synchronization,distributed-system
I'm just a little bit confused of this concept. I heard the words "Distributed system" a lot, but I'm not really sure my stuff is kind of "Distributed system". Basically, we have a master server( a very big one) as the front line production server. Then , in order to...
java,amazon-web-services,distributed,distributed-computing,amazon-sqs
I have a process that sends SQS messages every minute. It's important that the messages go out every minute so I'm planning on running the process on multiple instances so that it's more fault tolerant. Even though it's running on multiple instances I only want the SQS messages to go...
distributed-computing,hazelcast
I'm using Hazelcast 3.3. One member writes entries to an IMap and calls map.executeOnEntries(myEntryProcessor). The task of EntryProcessor is to just print the entries on console. However, the members (3 other and the 1st one = 4 members) seem to print overlapping set of entries. My understanding was that the...
distributed-computing,zookeeper,distributed-system,curator
I'm new to curator and zk - and wanted to double check my understanding with the rest of the community. It seems that documentation for curator is not that well covered. Are curator's persistent ephemeral nodes basically ephemeral znodes, but have extra mechanisms to re-establish connections once it's disconnected? Are...
distributed-computing,zookeeper,service-discovery,consul,consensus
I'm looking at consensus-type tools like ZooKeeper, Consul and Eureka and they all seem to market the same set of solutions: Service discovery Dynamic, centralized configuration management Synchronization primitives Consensus algorithms However the more I read about these things, the more I struggle to see how service discovery is really...
c#,.net,math,random,distributed-computing
I have a situation in which I must generate a random number, this number must be either zero or one So, the code is something like this: randomNumber = new Random().Next(0,1) However, the business requirements state that there is just 10% probability that the generated number is zero and 90%...
postgresql,distributed-computing,mesos,mesosphere
Recently I've discovered such a thing as a Apache Mesos. It all looks amazingly in all that demos and examples. I could easily imagine how one would run for stateless jobs - that fits to the whole idea naturally. Bot how to deal with long running jobs that are stateful?...
algorithm,math,distributed-computing,distributed-algorithm
I'm trying to find an answer to a problem in my Distributed Algorithms course, and to do so I want to get something clarified. What is the diameter of a graph with one node, with an edge to itself? Is it 1 or 0? If you are interested, the question...
ipython,mpi,distributed-computing
I want to use IPython's MPI abilities with distributed computing. Namely I would like MPI to be run with a machine file of sorts so I can add multiple machines. EDIT: I forgot to include my configuration. Configuration ~/.ipython/profile_default/ipcluster_config.py # The command line arguments to pass to mpiexec. c.MPILauncher.mpi_args =...