azure,logging,partitioning,azure-table-storage , How to partition Azure tables used for storing logs

How to partition Azure tables used for storing logs


Tag: azure,logging,partitioning,azure-table-storage

We have recently updated our logging to use Azure table storage, which owing to its low cost and high performance when querying by row and partition is highly suited to this purpose.

We are trying to follow the guidelines given in the document Designing a Scalable Partitioning Strategy for Azure Table Storage. As we are making a great number of inserts to this table (and hopefully an increasing number, as we scale) we need to ensure that we don't hit our limits resulting in logs being lost. We structured our design as follows:

However, in practice we have found this is less than ideal, because when we look at our logs at a glance we would like them to appear in order of time. We instead end up with blocks of results grouped by context, and we would have to get all partitions if we want to order them by time.

Some ideas we had were

As this is such a common application of Azure table storage, there must be some sort of standard procedure. What is the best practice for partitioning Azure tables that are used for storing logs?

Solution constraints


I have come across similar situation you encountered, based on my experience I could say:

Whenever a query is fired on an azure storage table, it does a full table scan if a proper partition key is not provided. In other words, storage table is indexed on Partition key and partitioning the data properly is the key to get fast results.

That said, now you will have to think on what kind of queries you would fire on the table. Such as Logs occurred during a time period, for a product etc.

One way is to use reverse ticks up to hour precision instead of using the exact ticks as part of Partition Key. That way an hour worth of data can be queried based on this partition key. Depending on the number of rows which fall in to each partition, you could change the precision to a day. Also, it will be wise to store related data together, that means data for each product would go to a different table. That way you can reduce the number of partitions and number of rows in each partition.

Basically, ensure that you know the partition keys in advance (exact or range) and fire queries against such specific partition keys to get results faster.

To speed up writing to table, you can use Batch Operation. Be cautious though as if one entity on the batch fails whole batch operation fails. Proper retry and error checking can save you here.

At the same time, you could use blob storage to store lot of related data. The idea is to store a chunk of related serialized data as one blob. You can hit one such blob to get all the data in it and do further projections on the client side. For example, an hour worth of data for a product would go to a blob, you can devise a specific blob prefix naming pattern and hit the exact blob when needed. This will help you get your data pretty fast rather than doing a table scan for each query.

I used the blob approach and have been using it for couple of years with no troubles. I convert my collection to IList<IDictionary<string,string>> and use binary serialization and Gzip for storing each blob. I use Reflection.Emmit based helper methods to access entity properties pretty fast so serialization and deserialization doesn't take a toll on the CPU and memory.

Storing data in blobs help me store more for less and get my data faster.


What should 'git log ^X' show?

$ git log --oneline 05f1c3b master: Modified 1.txt f7433e0 Updated by master e30dff9 Modified 1.txt 82ebf7a Modified 1.txt cbb2785 master: Added 4.txt after rebase branch was created 980a5d6 b1: Added 1.txt badc046 Resolved merge conflict 47a2d78 Mods by b2 c3eb61e Added by b1 9378e3d 1ST commit The following shows nothing:...

What are correct permissions for Linux Apache2 PHP 5.3 log file?

I discovered the reason why I was not getting entries into my php_errors.log file related to permissions. Right now, I have set it to 666 (rw-rw-rw-) but surely this is a security weakness? Thus, my question. php.ini file: error_log /var/log/httpd/php_errors.log log_errors On # ls -ld /var/log /var/log/httpd /var/log/httpd/php_errors.log drwxr-xr-x 6...

TelemetryClient does not send any data unless Flush is called

I'm using TelemetryClient (v0.17.0.576) directly in my code and it looks like I can push data to Azure only when I manually call Flush at the end which feels wrong. Am I missing something here ? var configuration = TelemetryConfiguration.CreateDefault(); configuration.InstrumentationKey = "KEY"; var client = new TelemetryClient(configuration); for (int...

C++ Have lone executable log exceptions?

I'm currently in a beta testing phase of my C++ Server. I'm sending it out to users to stress test it to see if an exceptions show up. However, since the .exe is a standalone, how do I make it so it automatically creates a dump or a log of...

Is there any sdk to log exceptions, events and errors in production app?

I want to catch all the un expected/undesirable events in my app during production. I want something which can send a error log file to server and I can easily read it through a web application. Do you know anything of this sort ?

Logging with log4j2 in Spring Boot app

I am trying to learn Spring Boot. But I completely mess up with a loggers dependencies. I have a simple pom: <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> <!-- Logging --> <dependency>...

node.js winston logger no colors with nohup

We are using winston logger in our project with the following transport settings: file: { filename: __base + '/log/server.log', colorize : true, timestamp : true, json : false, prettyPrint : true } If the application is started with nohup, log file is not colorized. It works only without nohup. nohup...

How to block writes to standard output in java (System.out.println())

In Java, how to block code from writing to system out? My app calls a 3rd party library that spams logs by issuing numerous System.out.println() calls. I don't have legal rights to decompile and patch the library. I'm running Websphere 8.5.x Considered using System.setOut(PrintStream out), but that will effect the...

Handling service bus Message.Complete() exceptions

Consider the scenario, an Azure service bus with message deduplication enabled, with a single topic, with a single subscription and an application that is subscribed to that queue. How can I ensure that the application receives messages from the queue once and only once ? Here is the code I'm...

Azure, login to separate accounts with one email

I'm a developer that has an Azure account for my own dev stuff. I log into my dev account using [email protected] Next, I set up a client with their own Azure account, then invited myself via [email protected] and set myself as a co-administrator for the client's subscription. When I subsequently...

Scala slf4j dynamic file name

I just successfully added Grizzled-SLF4J logger to my project using this link But using this properties, there is no option to create dynamic file name: org.slf4j.simpleLogger.logFile = /tmp/myapp.log org.slf4j.simpleLogger.defaultLogLevel = info org.slf4j.simpleLogger.showDateTime = true org.slf4j.simpleLogger.dateTimeFormat = yyyy'/'MM'/'dd' 'HH':'mm':'ss'-'S org.slf4j.simpleLogger.showThreadName = true...

Azure - Enable diagnostics for worker/web role

I would like to log every exception that occurs, (MVC cloudapp), in my table storage. I followed an official tutorial from Microsoft but it seems like they just complicate things too much. Tutorial can be viewed here: It just feels like it's too much of a hassle just because...

How to expose existing REST API through Azure Service Bus (or through something else)

I have an existing on-premise REST API from an external vendor. I'd like to expose this API unmodified to the outside world through an Azure website. So I have customers that run this API on-premise and I'm developing a PaaS/SaaS app that should access these on-premise API's. I also have...

connect to mysql database which is in ubuntu server

I am using below code to connect MySQL database in PHP. try { shell_exec("ssh -f -L 3307: [email protected]_ip sleep 60 >> logfile"); $this->_conn = $this->dbh = new PDO('mysql:host=;dbname=my_db', DB_USER, DB_PASS); $this->dbh->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION); } catch (PDOException $e) { die("Couldn't connect to database. Please try again!"); } I want to direct connect...

Azure NextMarker

I have created a container in Azure called files. It has 3 images and 2 txt files. With the URL listed above, NextMarker will be empty. If I use the following URL.. It shows the NextMarker field populated, and what I understand from this article is that using...

A parameter cannot be found that matches parameter name 'MaxSizeGB'

New-AzureSqlDatabase command on execution suddently started failing with parameter not found message. MSDN link mentions the parameter as supported. Command : New-AzureSqlDatabase Input PS C:\scripts>$database1 = New-AzureSqlDatabase -ServerName "lpqd0zbrseg" -DatabaseName "Database1" -Edition "Business" -MaxSizeGB 50 -Collation "SQL_Latin1_General_CP1_CI_AS" Output Error New-AzureSqlDatabase : A parameter cannot be found that matches parameter...

How can I read the logExtFileFlags from the “Default” site?

I am attempting to read IIS 7 configuration data remotely, specifically the logExtFileFlags from the Default site. When I look at the applicationHost.config on the remote server, I see the information in question: <system.applicationHost> <sites> <site name="Default" id="1" serverAutoStart="true"> <logFile logExtFileFlags="Date, Time, ClientIP, UserName, SeverIP, Method, UriStem, URIQuery, HttpStatus, TimeTaken,...

Application for viewing Azure service bus dead letters

I've been looking around the web and GitHub for an off-the-shelf dead letter viewer for Azure service bus. This is to allow our DevOps team to monitor, view and report on any dead letters for each subscription for each topic on our bus. I thought this would be a common...

Control access to web apps on Azure

I currently have numerous web apps which are just webapi's on Azure. I have a gateway web app that is accessible publically as it has its own authentication mechanisms. However the other web apps I don't want to be accessed publically, only the gateway web app should have access to...

Azure Mobile Service requires authentication for SignalR HTML/Javascript Client

I have a fresh Azure Mobile Service running locally at localhost:52253. I installed the latest SignalR Nuget Package: Azure Mobile Services .NET Backend SignalR Extension 1.0.450 I also have a test HTML/Javascript client served from localhost:54697/ After working out the CORs issues, I still cannot connect to signalR hubs because...

It was not possible to connect to the redis server(s); to create a disconnected multiplexer

I have the following piece of code to connect to azure redis cache. public class CacheConnectionHelper { private static Lazy<ConnectionMultiplexer> lazyConnection = new Lazy<ConnectionMultiplexer>(() => { return ConnectionMultiplexer.Connect(SettingsHelper.AzureRedisCache); }); public static ConnectionMultiplexer Connection { get { return lazyConnection.Value; } } } And I use it this way public static List<Models.Module>...

Distributed session implementation detail

With the reference to the structure of session module in ASP.NET below: As I understand, when the application uses distributed session provider (.e.g. Redis in Azure), the SessionStateModule will deserializes the user's session data into Session dictionary at the beginning of a request. What I'm wondering is whether the module...

Azure Mobile Services: migrate to non-Azure Windows Server,azure,azure-mobile-services
I am planning to use Windows Azure Mobile Services for a new project. As far as I am concerned, Mobile Services are closely tied to Windows Azure. So far, I'm pretty happy with Windows Azure. What if, in the future, I decide to port the functionality to a non-Azure Windows...

PHP MSSQL Database Azure

I am trying to get parse data from the SQL Database from Azure into a JSON using PHP. I have the php script on a free web hosting server. I recieve an ERROR when I am to connect to the Azure's SQL Database. My PHP Script <?php $serverName = ",1433";...

viewing google app engine Python logging messages in CodeEnvy

I'm trying to move my GAE development to the cloud. So far Codeenvy has the richest toolset however I'm struggling with one small issue. when I use the python logging library I don't know where to view these messages! def post(self): self.response.write('Processing form data...') feedback = self.request.get('content') I assumed...

log4j creates log file but does not write the messages that I am creating

Log file is getting created and logs are getting written into it.But , it doesn't write the things that i specify in LOG.trace("") and LOG.error("") etc. log4j.rootLogger=INFO,R # Direct log messages to stdout log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.Target=System.out log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss}%-5p %c{1}:%L - %m%n log4j.appender.R=org.apache.log4j.RollingFileAppender...

Different output when run on Azure than on local build

I have a C#, .NET MVC 4.5 web app with an action method which calls out to a web service with a query string parameter. Something like "/home/[email protected]" I'm using Encoding.UTF8 and my return code looks like this: return Content(responseText); This outputs as: 6^.a308666-86 which is correct. However, when I...

Log4j Implicit String Formatting

I am using log4j v1.2.14 for logging in my project and I am also using Java 7 String.format() to put variables in my output. Currently I am writing"Your var is [%s] and you are [%s]", myVar, myVar1)); Is this really the best way to output strings? I feel that...

website creation fails with serverfarm not found Azure Powershell

I am struggling to get the Drupal Webapp created using powershell with a GalleryTemplate Json file. It creates Serverfarm, Database servers but fails for website creation. I am not able to figure out how to move ahead as website creation need to use the created hostingplan serverfarm.It would be great...

monitor log4j behaviour under load

I want to test my J2EE application under high load of sessions accessing different pages. This web application uses Log4J to log bunch of errors,warnings and infos. I want to test what is the side effect of this load on writing log files, especially concurrent I/O writing actions. I found...

Unable to publish to Azure after 2014 update,,azure,sql-azure
I have an MVC project I have been publishing to azure for a while. I recently change laptop and moved to SQL Server 2014. Now when I try to publish I get 3 errors. The fist one looks to be the root course: Error 3 .Net SqlClient Data Provider:...

How can I add a .props file to a C# project?

I am trying to set up continuous delivery of Azure Web Jobs, and came across this post from the Azure blog. One of the steps there is to add a .props file to the web job application's properties. I am having difficulty adding a .props file. My web searches indicate...

Azure Active Directory Graph Client 2.0 - Contains Expression

When querying Users etc using the Azure ActiveDirectoryClient it does not seem possible to construct a expression that has the 'Contains' keyword e.g. activeDirectoryClient.Users.Where(u => u.Surname.Contains(searchString)) .ExecuteAsync() I can successfully use u.Surname.StartsWith or EndsWith. Is this supported? If not is there an alternative? Thanks....

Kestrel Running which framework?

I have a new ASP.NET 5 project and setup the project.json as below; "frameworks": { "dnx451": { "dependencies": { "ExternalLibrary": "1.3.0" } }, "dnxcore50": {} } I'm running the project through Kestrel with the default command defined in project.json. "kestrel": "Microsoft.AspNet.Hosting --server Kestrel --server.urls http://localhost:5001" Everything works fine. The project...

Setup Codeigniter For Windows Azure

I've been trying to Setup Codeigniter 3 on the Windows Azure platform,everything seems fine for the normal setup but when i uploaded my application(Which i did via github so all the files are intact) and my session is set to autoload from config,i get the error "The resource you are...

Spark streaming on YARN executor's logs not available

I'm running the following code .map{x => Logger.fatal("Hello World") x._2 } It's spark streaming applciation runs on YARN. I upadted log4j and provided it with spark-submit (using --files). My Log4j configuration was loaded which I see from logs and applied to Driver's logs (I see my log level only and...

setting up azure ad certificate auth using powershell

I am attempting to setup an Azure AD application to authenticate using a certificate. I have been using this guide to successfully set this up after failing following this guide. I wish to now automate the process so it can be added to our general build scripts, looking at the...

Error deploying database to SQL Azure

I'm using forms authentication to handle users and attempting to deploy my database to SQL Azure, but getting this error message: The only table of the four listed that I utilize is aspnet_Membership, and the only other table I use is aspnet_Users from implementing forms authentication. What is TextInRowSize and...

Python root logger messages not being logged via handler configured with fileConfig

The Problem: Given a logging config and a logger that employs that config, I see log messages from the script in which the log handler is configured, but not from the root logger, to which the same handler is assigned. Details: (Using Python 2.7) I have a module my_mod which...

View local Azure Webjobs SDK dashboard for debugging WebJobs SDK console process

I have a VS 2013 RC 4 Azure Webjobs project that I'm currently debugging. The project is a console project and has a function that uses a QueueTrigger attribute. I get the following errors in the console output when I run the job and insert a message to the template-mail-requests...

Logging all network traffic in Spring mvc

I have spring mvc application using RequestBody and ResponseBody annotations. They are configured with MappingJackson2HttpMessageConverter. I also have slf4j set up. I would like to log all json as it comes in and out from my controller. I did extend MappingJackson2HttpMessageConverter @Override public Object read(Type type, Class<?> contextClass, HttpInputMessage inputMessage)...

How can I view Azure internal errors?,azure
After uploading my site to Microsoft Azure, server returns this text instead of my site: "The page cannot be displayed because an internal server error has occurred." Where can I see what's the problem?...

Logging operations in lightadmin

I have a Java web application which use Spring and Hibernate and I plan to use lightadmin to provide an administration interface. However, I found very little information about the logging part of lightadmin : if I have such an adminsitration interface, I would like that any operation made to...

Azure - is one 'block blob' seen as one file?

Question background: This may be a simple question but I cant find an answer to it. I've just started using Azure storage (for storing images) and want to know if one 'blob' holds a maximum of one file? This is my container called fmfcpics: Within the container I have a...

SimpleInjector decoration with additional dependency

I have a decorator SomethingLoggerDecorator that is supposed to decorate ISomething instances with logging: public class SomethingLoggerDecorator : ISomething { private readonly ISomething decoratee; private readonly ILogger logger; public SomethingLoggerDecorator(ISomething decoratee, ILogger logger) { this.decoratee = decoratee; this.logger = logger; } public void DoSomething() { this.logger.Info("Doing Something"); this.decoratee.DoSomething(); } public...

Java.util.logger new file every day

I am using java.util.logging framework to implement logging in my JSF application. I have successfully done implementing this, but however I have a new requirement to rotate the logs and create a new log file for each day. I am not able to figure out how this can be implemented....

How to get the logs in XML file in cleartool. I want the output of command cleartool lshistory in XML file

I am trying command cleartool lshistory > file.txt through this I am getting the file in text format similarly I want the log file in XML format.

Is there a way to implement a wcf service connected to a database without paying for azure?

It seems like everything that I read is saying that for Windows Phone 8 Development, in order to create a database that connects to my WCF Service, I need to pay for it (Azure). Is that the truth? I want to create a database that my mobile app can access...

Domain redirection to Azure subscription

I have one website running under current domain which is not azure site. we build a new application with the new azure subscription I wanted to redirect/host the to my azure subscription. how can we achieve this??...