FAQ Database Discussion Community


How to reproduce experiments in a non-adhoc manner

distributed,distributed-computing,experimental-design
For a master thesis in the field of distributed computing (think of hadoop and two-level schedulers like Mesos) I'm setting up various experiments on a university cluster. However I'm already piling up bash scripts which function as a driver for the experiment. I miss composability and reuse between subparts of...

mongodb shards, how many mongod for this case

database,mongodb,nosql,distributed
In mongodb. If you want to build a production system with two shards, each one a replica set with three nodes, how may mongod processes must you start? why the answer is 9?...

Synchronize actions in a distributed system

distributed,distributed-computing,zookeeper,distributed-system
What techniques/tools can be used to implement a distributed system with these requirements: At a given time, the system can be in one of 3 states: SYNCING, COMPUTING, or IDLE. Each node in the system can receive two instructions: sync() and compute(). A sync() instruction will be sent to all...

How does etcd handle reads/writes during a network partition?

distributed,etcd
I am looking for something to use as a simple service registry and am considering etcd. For this use-case availability is more important than consistency. Clients must be able to read/write keys to any of the nodes even when the cluster is split. Can etcd be used in this way?...

Why doesn't LocateRegistry.getRegistry() fail if I specify the wrong port number?

java,rmi,distributed,distributed-computing
I'm trying to distribute a simple room booking system that I implemented, I tried researching and this is what I came up with.. but When I run them and enter different port numbers the client still runs..shouldn't it only run if the port number matches the one I entered for...

How to execute a method once (multiple processes/instances) per minute utilizing AWS

java,amazon-web-services,distributed,distributed-computing,amazon-sqs
I have a process that sends SQS messages every minute. It's important that the messages go out every minute so I'm planning on running the process on multiple instances so that it's more fault tolerant. Even though it's running on multiple instances I only want the SQS messages to go...

Aggregating distributed logfiles into one logfile on real-time

monitoring,distributed,logfile
I got in legacy a system which save its logfiles on 15 different servers during 1 job run. Now, there is a need to aggregate all of them into 1 file, and just as important- on real-time. I started by ssh through python, but its not answering the real-time need....

distributed algorithms simulation in erlang

algorithm,erlang,distributed
I'm a beginner with Erlang. I would like to use it to observe the execution of "textbooks" distributed algorithm (leader election, consensus...)for pedagogic purpose. At that stage, I describe the topology of my system as a graph (dict from int to list of ints) and based on that, i instanciate...

Distributing jobs over multiple servers using python

python,ipython,distributed
I currently has an executable that when running uses all the cores on my server. I want to add another server, and have the jobs split between the two machines, but still each job using all the cores on the machine it is running. If both machines are busy I...

Hbase on hadoop not connecting on distrubuted mode

hadoop,hbase,bigdata,ubuntu-14.04,distributed
Hi I AM TRYING TO SETUP HBASE(hbase-0.98.12-hadoop2) ON HADOOP(hadoop-2.7.0) Hadoop is running on localhost:560070 its running fine . my hbase-site.xml as show below <configuration> <property> <name>hbase.rootdir</name> <value>hdfs://localhost:9000/hbase</value> </property> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> <property> <name>hbase.zookeeper.quorum</name>...

Convert Matrix to RowMatrix in Apache Spark using Scala

apache,scala,matrix,apache-spark,distributed
I'd really like to convert my org.apache.spark.mllib.linalg.Matrix to org.apache.spark.mllib.linalg.distributed.RowMatrix I can do it as such: val xx = X.computeGramianMatrix() //xx is type org.apache.spark.mllib.linalg.Matrix val xxs = xx.toString() val xxr = xxs.split("\n").map(row => row.replace(" "," ").replace(" "," ").replace(" "," ").replace(" "," ").replace(" ",",").split(",")) val xxp = sc.parallelize(xxr) val xxd = xxp.map(ar...

Centralized/Distributed/Service oriented Architecture/Application

service,architecture,distributed,centralized
I am doing a system architecture and my knowledge from college doesn't help me when it comes to understand the subtle differences between centralized, distributed and service oriented architecture/application. If I take a typical client/server architecture, the client sends requests to a server, the server then sends responses to the...

Contradiction in Lamport's Paxos made simple paper

algorithm,distributed,distributed-system,paxos,consensus
Phase 2. (a) If the proposer receives a response to its prepare requests (numbered n) from a majority of acceptors, then it sends an accept request to each of those acceptors for a proposal numbered n with a value v, where v is the value of the highest-numbered proposal...

How to stop mesos from offering resources to a framework?

distributed,distributed-computing,mesos,mesosphere
I have a use case where I have 20-30 frameworks runnings on mesos cluster that has over 200 nodes. A lot of the times mesos is offering resources to frameworks that do not want any offers at all. While doing that, it is offering little resources to frameworks that actually...

How are distributed queues architectured?

queue,distributed,distributed-computing,distributed-system
What are architectural patterns/solutions that make distributed queues tick? Please share for both ordered and non-ordered types....

OrientDB doesn't save edge in/out and properties

distributed,orient-db
I've installed orientdb in distributed mode but I have a problem during creation of the edges (lightweightmode is disabled). When I create an edge everything seems to work fine but actually orientdb doesn't save the properties and even the link between out-in objects! I run the server executing dserver.sh. If...

How to include “start orbd -ORBInitialPort 1050” in java?

java,distributed,corba
I am trying to learn how to use CORBA by example in this web site http://www.cs.mun.ca/java-api-1.5/guide/rmi-iiop/rmiiiopexample.html So in this example usually I should run command line to run orb start orbd -ORBInitialPort 1050 Is there anyway to include this in java program?...

Does Data Synchronization between servers count as distributed system? [closed]

database,distributed,distributed-computing,data-synchronization,distributed-system
I'm just a little bit confused of this concept. I heard the words "Distributed system" a lot, but I'm not really sure my stuff is kind of "Distributed system". Basically, we have a master server( a very big one) as the front line production server. Then , in order to...

Logical Clocks: Lamport Timestamps

messaging,distributed,clock,timing,distributed-system
I am currently trying to understand Lamport timestamps. Consider two processes P1 (producing events a1, a2,...) and P2 (producing events b1, b2,...). Let C(e) denote the Lamport timestamp associated with event an e. I created timestamps for each event as described in the Wikipedia article about Lamport timestamps: According to...

The reduce task is stopped by Too Many Fetch Failure message in Hadoop multi node (10x) cluster

java,linux,ubuntu,hadoop,distributed
I am using Hadoop 1.0.3 for a 10 Desktop cluster system each having Ubuntu 12.04LTS 32 bit OS. The JDK is 7 u 75. Each machine has 2 GB RAM with core 2-duo processor. For a research project, I need to run a hadoop job similar to "Word Count". And...