go,redis , Go Redis Loading


Go Redis Loading

Question:

Tag: go,redis

I am trying to load 200 million keys into redis and usually start to get an error at around 31 million keys and have to stop.I am using golang and the redis library "github.com/garyburd/redigo/redis"

I set up a connection pool as so:

func newPool(server string) *redis.Pool {
    return &redis.Pool{
        MaxIdle: 3,
        MaxActive: 10,
        IdleTimeout: 240 * time.Second,
        Dial: func () (redis.Conn, error) {
            c, err := redis.Dial("tcp", server)
            if err != nil {
                return nil, err
            }
            return c, err
        },
        TestOnBorrow: func(c redis.Conn, t time.Time) error {
            _, err := c.Do("PING")
            return err
        },
    }
}

I then try to fill up redis with values with this function:

func RedisServerBatchLoadKeys(rtbExchange string, keys []string){
  redisLock.Lock()
  defer redisLock.Unlock()
  retry := 0
  for {
    conn := GetConnOrPanic(rtbExchange)
    defer conn.Close()
    conn.Send("MULTI")
    for _, key := range keys {
      conn.Send("SET", key, maxCount)
      conn.Send("EXPIRE", key, numSecondsExpire)
    }
    _, err := conn.Do("EXEC")
    if err == nil {
      break
    } else if !(err == io.EOF) {
      CheckRedisError(err, rtbExchange, "Could not load batch")
    } else {
      retry ++
    }
    if retry >= 10 {
      CheckRedisError(err, rtbExchange, "Could not load batch - 10 retries")
    }
  }
}

I have been getting numerous errors such as:

Am I doing something fundamentally wrong or do I have to add in more error checks (aside from the EOF that I added).

Thanks,


Answer:

Just a guess: 200 million keys is a lot. Do you have enough memory for that size database?

The Redis docs say:

Redis can handle up to 2^32 keys, and was tested in practice to handle at least 250 million of keys per instance.

In other words your limit is likely the available memory in your system.

They also say:

What happens if Redis runs out of memory?

Redis will either be killed by the Linux kernel OOM killer, crash with an error, or will start to slow down.

It seems plausible to me that you're not able to connect because the server is actually down. Perhaps it gets restarted, and the next time you run your script it gets to the same place every time because that's when you run out of memory.

If this is your problem there are a couple things you could try:

  1. Use a redis hash which can store data more efficiently. See http://redis.io/topics/memory-optimization
  2. Partition (shard) your data set across multiple servers (for example if you had 4 servers you could take your key % 4 to determine which redis server to store under) If what you're going for is O(1) lookup you'll still get that, though you've made your system more brittle because there are multiple points of failure.

Related:


Golang, how to return in func FROM another func?


go,return,func
I want to end execution in parent func apiEndpoint() upon calling/exiting in a child func apiResponse() func apiEndpoint() { if false { apiResponse("error") // I want apiResponse() call to return (end execution) in parent func // so next apiResponse("all good") wont be executed } apiResponse("all good") } func apiResponse(message string)...

Find the shortest path sum in a matrix. Is Dijkstra not optimal for this case?


algorithm,go
I am trying to solve the following problem from project euler (please take a look at description and the example in the link, but here is the short explanation). in the matrix, find the minimal path sum from the top left to the bottom right, by moving left, right, up,...

Unmarshal JSON into a map in Go


json,dictionary,go
I'm having trouble figuring out how to load a "subsection" of JSON file into a map element. Background: I'm trying to unmarshal a somewhat complicated configuration file which has a strict structure, so I assume it's preferable to unmarshal into a "static" structure rather than into an interface{}. Here's a...

Does Mgo cache connection strings?


mongodb,go,mgo,compose
My Go application has been connecting to a MongoDB (hosted on Compose.io) using MGO with no issues. Today I decided to delete this database and add a different one (again using Compose). I updated the connection string. So I connect using: db, err := mgo.Dial("mongodb://<username>:<password>@dogen.mongohq.com:10048/db-name") with the username and password...

Golang ORDER BY issue with MySql


mysql,database,go
I can't seem to dynamically ORDER BY with db.Select(). I've Googled without any luck... WORKS rows, err := db.Query("SELECT * FROM Apps ORDER BY title DESC") DOES NOT WORK rows, err := db.Query("SELECT * FROM Apps ORDER BY ? DESC", "title") I'm not getting any errors, the query simply fails...

convert chan to non chan in golang


go,type-conversion
Is it possible to let function funcWithNonChanResult have the following interface: func funcWithNonChanResult() int { And if I want it to use function funcWithChanResult with the interface: func funcWithChanResult() chan int { In other words, can I somehow convert chan int to int? Or I must have chan int result...

How do I synchronize a variable number of channels in Go?


go
I am testing out concurrency in Go by writing a client (in the form of a web server) that makes many requests to a web server and then returns how long it took to make those requests. Basically a benchmark tool. Here is to code I am using currently: main.go...

Serving Static Files with a HTTP 500 Status


go
Is there a way to serve static files over HTTP in Go with a custom status code (without re-writing a significant amount of private code)? From what I can see: http.ServeFile calls the helper function http.serveFile It then calls http.ServeContent after determining the mod time and size of the file/dir...

Marshalling empty map[string]interface{} results in “null” instead of nil


go
I have a problem where Go is inserting "null" into my PostgreSQL database's jsonb columns if I try to do the following, and the structs property (in this case of type map[string]interface{}) is empty: accessMembers, _ := json.Marshal(c.AccessMembers) Doing a test print it outputs the same as the value which...

nodejs Kue job processing logic


node.js,redis,kue
I have a very simple logical question. I will be running job processing logic on a separate app server. My job processing app will be a standalone app, doing nothing just processing jobs. In my code, how do I make sure that my app continuously keep checking redis server for...

Although maps are always reference types, what if they're returned from a non-pointer receiver?


pointers,dictionary,go
Supposedly maps are reference types in Go, so when returning them from functions, you don't need to pass as a pointer to the map in order for the changes to be visible outside the function body. But what if said map is returned from a method on a non-pointer struct?...

Socket.io redis How data stored and cleared


redis,socket.io,socket.io-redis
i am hosting an app on heroku which is using socket.io. it is using sockets and i am using heroku 4 standard 1X dynos . So for this i used redistogo service and socket.io-redis plugin. it's working great but i want to know does socket.io-redis also clear the data from...

Getting a lost Sentinel error message for Redis


redis,spring-boot,spring-data-redis
I am running a spring boot service using spring data redis and here is the following configuration. The service seems to work but I am seeing a stream of Lost Sentinel messages in the logs. The sentinel nodes are reachable form the VM where I am running the service. I...

Set options for ZADD command in laravel redis


php,laravel,redis,set
I'm trying to set options for ZADD with laravel redis but am failing. The option I need to set is NX, as stated in the documentation: ZADD options (Redis 3.0.2 or greater) ZADD supports a list of options, specified after the name of the key and before the first score...

panic: runtime error: invalid memory address or nil pointer dereference when running Blockchainr


go,blockchain
I run blockchainr and get the following terminal output: 17:39:57 2015-06-16 [INF] loading db leveldb panic: runtime error: invalid memory address or nil pointer dereference panic: runtime error: invalid memory address or nil pointer dereference [signal 0xb code=0x1 addr=0x0 pc=0x402cb63] goroutine 1 [running]: main.search(0x4911ef8, 0xc20806e2d0, 0x0, 0x0, 0x4911ef8) /Users/mesquka/Downloads/blockchainr-master/src/blockchainr/main.go:185 +0x3fa...

Does exist on Go something like macros in C++ like #ifdef so I can choose what to build based on flag?


go
I need to build in go for linux and windows with different packages same file on windows I need to import github.com/hashicorp/go-syslog and on linux import log/syslog and inside code in file I have to use syslog.ALERT or gsyslog.ALERT depending on os. Does exist on Go something like macros in...

Why using naked return and the normal return give me different results?


go,rot13
I'm playing around with Golang tour and I wonder why using naked return give me the correct result but the normal one doesn't. This is the exercise that I have this problem https://tour.golang.org/methods/12. The objective is to create a reader that can decipher rot13. and the rot13 function is already...

Golang - using/iterating through JSON parsed map


json,parsing,go
With PHP and Javascript (and Node) parsing JSON is a very trivial operation. From the looks of it Go is rather more complicated. Consider the example below package main import ("encoding/json";"fmt") type fileData struct{ tn string size int } type jMapA map[string] string type jMapB map[string] fileData func parseMapA(){ var...

How To Restore a Missing Redis Service


linux,redis,centos
I installed an older version of Redis on a CentOS server. I tried to remove that old version and update it to latest version, but it seems that the redis service is gone and the new version installation doesn't reproduce it. Is there any way I can uninstall the Redis...

How to keep a strong reference in Go?


arrays,go,slice
Is there any way I can keep a strong reference in Go? Given the following convoluted piece of code: package main import ( "fmt" ) func main() { slice := make([]int, 5) slice[3] = 25 // whatever index between 0 and 4 included I don't care slicesArray := make([]*[]int, 2)...

JSON decoding in golang


json,curl,go
So I tried something based on the example here in my code, and get no data, but no error either. The code is: import ( "io" "fmt" "net/http" "encoding/json" ) type Credential struct { username string `json:"username"` password string `json:"password"` } func login(res http.ResponseWriter, req *http.Request) { if req.Method ==...

Go Yaml Interpretation Example


go,yaml
This Go Yaml Interpretation Example, https://gist.github.com/suntong001/74c85d15b19ef4b14b0e, can unmarshal a simple YAML like this: foo: 1 bar: - one - two Now I want to interpretation an array of the exact same data structure, what's the correct way to do? Below is my code so far, and it is not working...

CGO converting Xlib XEvent struct to byte array?


c,go,xlib,cgo
I am creating a simple window manager (code based of the c code in tinywm) in Golang. To use Xlib, I am using cgo, so my header is: // #cgo LDFLAGS: -lX11 // #include <X11/Xlib.h> And I have a variable declaration, like: event := C.XEvent{} And then, I use this...

Golang failes to change the name of imported module


go,docker,skydns
I'm trying to bring SkyDNSv1 back to life and build it from my fork (here is Dockerfile). SkyDNS was really good and simple tool for the quick service discovery, but it wasn't updated for a long time. There is an error in build process and it's caused by third party...

image.Decode results in “unknown format” when source is multipart.File


image,go,martini
I have a multipart.File that a user uploads to my server, and then I take that file and upload it to s3 using aws-sdk-go, but I also want to create a thumbnail of that image. The code below works fine in my tests when I file is the return value...

Why won't mgo unmarshall my struct properly?


mongodb,go,bson,mgo
Earlier I posted this question asking about writing custom BSON marshalling/unmarshalling in Go using mgo. Now I've come to test it I think I've hit on a bigger problem. All my structs unmarshal to nil values. This is my currency struct with the implementations of bson.Getter and bson.Setter: type Currency...

Handling Custom BSON Marshaling (Golang & mgo)


json,mongodb,go,bson
I have a number of structs that require custom marshalling. When I was testing I was using JSON and the standard JSON marshaller. As it doesn't marshal unexported fields, I needed to write a custom MarshalJSON function, which worked perfectly. When I called json.Marshal on the parent struct containing the...

Different performances in Go slices resize


go,stack,benchmarking,slice
I'm spending some time experimenting with Go's internals and I ended up writing my own implementation of a stack using slices. As correctly pointed out by a reddit user in this post and as outlined by another user in this SO answer Go already tries to optimise slices resize. Turns...

Correct use of go context.Context


concurrency,go,gorilla
I just read the article: Build You Own Web Framework In Go and for sharing values among handlers I picked the context.Context and I'm using it in the following way to share values across handlers and middlewares: type appContext struct { db *sql.DB ctx context.Context cancel context.CancelFunc } func (c...

Why does my concurrent function exit prematurely in Go?


concurrency,go
I am looking through Go Bootcamp and am reading the Go Concurrency chapter right now. I have never used concurrency before in programming and don't understand the output of this program: package main import ( "fmt" "time" ) func say(s string) { for i := 0; i < 2; i++...

Referencing yourself inside of a struct


struct,go
Let's say I have a type that's a struct like so: type Authorization struct { Username string Password string Handler func(http.HandlerFunc) http.HandlerFunc } And I have an array of these: type Authorizations map[string]*Authorization I want to be able to do something like this: var auth = Authorizations{ "test": *Authorization{ "someusername",...

http.newRequest not sending post data


php,go
I have the following code to send post data to a server, but the server is not detecting any post data on the request. Client code: cookieJar, _ := cookiejar.New(nil) client := &http.Client{ Jar: cookieJar, } postUrl := os.Args[1] username := os.Args[2] password := os.Args[3] data := url.Values{} data.Set("username", username)...

Loop until i get correct user


ruby,redis
I have users stored in Redis and want to be able to call only certain subsets from a set, if i don't get the correct user back i want to put it back in the set and then try again until i get one of the desired users @redis =...

How to change a float64 number to uint64 in a right way?


go,floating-point,type-conversion,floating-point-conversion
package main func main() { var n float64 = 6161047830682206209 println(uint64(n)) } The output will be: 6161047830682206208 It looks like that when float64 change to uint64, the fraction is discarded....

How can I redirect, in Go, the stdout and stderr of a command to both the console and a log file while outputting in real time?


go,io,sync
The following bit of code does exactly what I want, except it only prints to the console. cmd := exec.Command("php", "randomcommand.php") cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr if err := cmd.Run(); err != nil { log.Fatal(err) } randomcommand.php: // randomcommand.php simply alternates output between stdout and stderr 20 times $stdout...

Declare variable with format specifier in Go


go
In Python, I can declare a variable as d ='/some/dir/%s' and later replace %s with any value as >>> d = '/some/dir/%s' >>> d % "hello" '/some/dir/hello' Is it possible to do the same in Go? If so, how?...

More strict compile options in GoLang


go
Is there an option in golang to fail build if a function returns a value and the returned value is not used? For example, I want an error / warning when we have: func abc() error and it is called without an lvalue as: abc() EDIT: This is not for...

Trouble retrieving Has Many using Gorm


go,go-gorm
When I try to get episodes back from a podcast I get invalid association []. Not sure what I'm doing wrong. package main import ( "log" "github.com/jinzhu/gorm" _ "github.com/mattn/go-sqlite3" ) type Podcast struct { Id int Title string RssUrl string `sql:"unique_index"` Url string Episodes []Episode } type Episode struct {...

Cannot call tests inside packages other than main


go
I am writing tests for my go code and inside sender folder, sender package I have added exposed_api_test.go ( also tried exposed_api_test.go because I have code in exposed_api.go) package sender import ( "log" "testing" "github.com/stretchr/testify/assert" ) func TestTimeConsuming(t *testing.T) { assert.Equal(t, "test", "test1") } and when I run build and...

Golang switch between structs


struct,go,interface,switch-statement
I'm new to golang and I'm trying to create a function that, based on the struct it's used on, will return a formatted string using Sprintf type Name struct { Title string First string Last string } type Location struct { Street string City string State string Zip string }...

Getting a PHP configuration variable in Go


php,go
I want to get the output of the command php -r 'echo get_cfg_var("some_var");' to check it against a predefined value. Currently, I have the following code: variableName := "default_mimetype" cmd := exec.Command("php", "-r", "'echo get_cfg_var(\"" + variableName + "\");'") out, err := cmd.CombinedOutput() after running, err.Error() returns "exit status 254"...

How to make this code more performant?


performance,go,hashmap
I have this snippet of code that iterate over a map and filter some fields based on relation type, I have to run two loops and have a feeling that it is going to be slow for big maps. Is there any way I can technique/refactoring to make this code...

Build a go binary for both Linux and Mac [duplicate]


go
This question already has an answer here: Cross compile Go on OSX? 2 answers How can I go build a project in a way that it works for both Mac and Linux (x86 and x64)? Is that even possible? I searched and found some stuff, but cannot make them...

Why doesn't “go get gopkg.in/…” work while “go get github.com/…” OK?


windows,git,powershell,github,go
I try to use go get gopkg.in/fatih/pool.v2 to install pool according to Readme.md, but can't success: C:\Users\xiaona\Documents\GitHub> go get -v gopkg.in/fatih/pool.v2 Fetching https://gopkg.in/fatih/pool.v2?go-get=1 https fetch failed. Fetching http://gopkg.in/fatih/pool.v2?go-get=1 import "gopkg.in/fatih/pool.v2": http/https fetch: Get http://gopkg.in/fatih/poo l.v2?go-get=1: dial tcp 107.178.216.236:80: ConnectEx tcp: A connection attempt failed because the connected party did not...

How do I get the submit value from the request object


html,go
I am trying to make a form POST request with two different submit buttons. I would like to try and get the submit button value in go. How do I go about getting this from the http.Request object. HTML Code: <form action="/save" method="POST"> <div><span>Title: </span><textarea name="title" placeholder="Link">{{printf "%s" .Title}}</textarea></div> <div>...

Mail with html content shows break lines or ignores newlines


email,templates,go,mandrill
I am sending mail to users via mandrill and I using both smtp and mandrill api to send. Content of the mail is rendered go template (.tpl) When I put template like Hi {{.name}}, <br/> This is support. <br/> it sends via mandrill api ok, but is visible when I...

measure execution time and stop waiting if too long in golang


go
I am trying to measure execution time of funcWithUnpredictiveExecutionTime function. func measureTime(expectedMs float64) (ok bool) { t1 := time.Now() funcWithUnpredictiveExecutionTime() t2 := time.Now() diff := t2.Sub(t1) The measuring is fine when funcWithUnpredictiveExecutionTime works faster than I expected. But if it works slower than expectedMs the measuring will not stop right...

Golang - structs, intializing and memcpy - what is the proper way?


arrays,struct,go,initialization,custom-type
I'm new to Go and I wish to translate some C/C++ code I have to Go but I haven't been able to. The problem resides in 2 places: how can I initialize a struct I've defined and how to do a "memcopy" The code I'm talking about is this: http://play.golang.org/p/e8N255qEAk...

Server initiated requests


http,go,http2
I know that http is a request-response protocol, the problem in short is a client makes a request to the server to start a long running process, and i want to inform the client over the progress with a simple json message containg progress info. in http1 i know that...

Redis: Delete user token by email ( find Key by Value )


node.js,express,redis
I have followed tutorial on how to create token-based authentication with node from this tutorial http://www.kdelemme.com/2014/08/16/token-based-authentication-with-nodejs-redis/ I got it all worked out, but I got 1 problem. The way I store token is : KEY = TOKEN VALUE = UserData (Username, email, etc.) To protect multiple devices login, I would...