mysql,sql,database,performance,csv , What would be the fastest way to insert this data


What would be the fastest way to insert this data

Question:

Tag: mysql,sql,database,performance,csv

Okay, so I have a MySQL table called entries which contains the columns name VARCHAR(255) NOT NULL and address VARCHAR(255)

The table has about a million sets of data.

For every set of data, name has a value e.g. "john" whilst address is NULL.

For example:

+------+---------+
| name | address |
+------+---------+
| john | NULL    |
+------+---------+
| jake | NULL    |
+------+---------+
| zach | NULL    |
+------+---------+

I received a CSV file which contains names along with their corresponding address in the format of name:address.

Like I said, the entries table has nearly a million entries, so the csv file has about 800,000 lines.

I want to take each line in the csv, and insert the address where the name is the same which would be:

UPDATE `entries` SET `address` = <address from csv> WHERE `name` = <name from csv>;

I made a Python script to open the csv file reading it line by line. For each line, it would store the name and address in separate variables. It would then execute the query above, but it was taking too long to insert the data into the columns.

Is there anyway I could do this in MySQL, if so, what is the fastest way?

Thanks.


Answer:

You can import the CSV file into a separate table using mysql LOAD DATA INFILE and then update the entries table using JOIN statement on the basis of similar column name.

E.g: update entries a inner join new_table b on a.name = b.name set a.address = b.address ;

Here new_table is imported from the CSV file..

Don't forget to add index on both tables for the name column so that it would be fast..


Related:


SQL Multiple LIKE Statements


sql,sql-server,tsql,variables,like
I'm currently working on a report that shows me all post codes covered by our sales team. Each team covers over 100 post codes. What i would like to do is create a report that brings back the clients within the post code. Currently my code looks like this. SELECT...

Fastest way to add a grouping column which divides the result per 4 rows


sql,sql-server,tsql,sql-server-2012
If i have a resultset like this for example (just a list of numbers) : 1,2,3,4,5,6,7,8,9,10,11 and I would like to add a grouping column so i can group them per 4 like this : 1,1,1,1,2,2,2,2,3,3,3 (The last one in this examle does not have a forth element, so that...

Temporarily Number Rows to Group by in MYSQL


mysql
I am trying to create a temporary column that gives each set of five results a number so that I can group by.. I am having a lot of trouble trying to work out how to do this. I don't want to take any criteria into account, I want the...

C# MySQL Parameters.AddWithValue


c#,mysql
Hello I found this code snippet for Adding Values to MySQL Commands in C# MySqlCommand command = connection.CreateCommand(); command.CommandText = "INSERT INTO tb_mitarbeiter (Vorname) VALUES (?name)"; command.Parameters.AddWithValue("?name", mitarbeiter); connection.Open(); command.ExecuteNonQuery(); Now I want to add data to more than one coloumn, but if i try it like this: command.Parameters.AddWithValue("?id", Projektid,...

Notice: Array to string conversion in “path of php file” on line 64


php,mysql,arrays,oracle
PHP CODE -: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <title>Untitled Document</title> </head> <body> <?php $rows =0; $fp = fopen("leave1.csv","r"); if($fp){ while(!feof($fp)){ $content = fgets($fp); if($content) $rows++; } } fclose($fp); //echo $rows; $_SESSION['rows'] = $rows; ?>...

Convert AWK command to sqlite query


sql,awk,sqlite3
I have to parse very big (abt 40Gb) text files(logs) very often. Usually AWK/grep is enough for my needs, but logs are growing and now I'm curious whether sqlite3 will allow me to do the same things more efficiently. I chosen sqlite for my tests as it installed out-of-the-box (SQLite...

I Want to fetch SQL Records in MySQL of current Year


mysql
I am using the query SELECT count(enq.`enquiryId`), Month(enq.`date`), Year(enq.`date`) FROM enquiry enq GROUP BY MONTH(enq.`date`) But I get all year records. I just want to get Current or any input year(At Runtime) records. Please help....

concatenate field names in a mysql update with inner join


php,mysql
Concatenating two field names from different tables using INNER JOIN during UPDATE statement. I am trying to concatenate two field names in a mysql update statement. This is what I have so far, needless to say it isn't working, any suggestions would be much appreciated. UPDATE products INNER JOIN sub_categories...

SQL varchar variable inserts question mark


sql,sql-server-2012
I've a database value that when inserted into a SQL variable, shows with question mark at the end !! can't find a reason?! declare @A varchar(50) = 'R2300529‏' select @A Results: R2300529? any explanation? i'm using SQL server 2012....

Query how often an event occurred at a given time


mysql,sql
[Aim] We would like to find out how often an event "A" ocurred before time "X". More concretely, given the dataset below we want to find out the count of the prior purchases. [Context] DMBS: MySQL 5.6 We have following dataset: user | date 1 | 2015-06-01 17:00:00 2 |...

Trying to rewrite mysql_* to pdo


php,mysql,pdo
First of all I should say that I started to learn PDO and trying to rewrite some old mine codes. Here is one that I found on tutorial for a jquery/ajax/pdo/mysql comment system which is working. This is the mysql_* part which I'm trying to rewrite -> file submit.php which...

MySQL - How can I know my query is tuned?


mysql,performance,explain
I'm creating a DB from scratch and I'm trying to create queries with performance in mind. For test purpose I filled my tables with test data. My query needs to join 3 tables: 2 with millions of rows and the third with hundred thousands rows. How can I know my...

MySQL substring match using regular expression; substring contain 'man' not 'woman'


mysql,regex
I have an issue while I fetch data from database using regular expression. While I search for 'man' in tags it returns tags contains 'woman' too; because its substring. SELECT '#hellowomanclothing' REGEXP '^(.)*[^wo]man(.)*$'; # returns 0 correct, it contains 'woman' SELECT '#helloowmanclothing' REGEXP '^(.)*[^wo]man(.)*$'; # returns 0 incorrect, it can...

T-SQL Ordering a Recursive Query - Parent/Child Structure


sql,tsql,recursion,order,hierarchy
I am trying (and failing) to correctly order my recursive CTE. My table consists of a parent-child structure where one task can relate to another on a variety of different levels. For example I could create a task (this is the parent), then create a sub-task from this and then...

Retrieve Values As Column


mysql,sql
I've the following table in my PHPMYADMIN The desired output is headline                              impressions              clicks Buy A new Iphone                 ...

MySQL Query returning strange values


php,mysql
The query is supposed to do the following: Obtain the question and associated answers based on the identification number of the question. (In this case the identification number is called account_id. Order them so that each question (and it's appropriate answers) are lined up in order. The query: SELECT *...

PHP / MySQLi: How to prevent SQL injection on INSERT (code partially working)


php,mysql,mysqli,sql-injection,sql-insert
I am new to PHP and hope someone can help me with this. I would like to store two values (an email and a password) in a MySQL db using PHP. The input is passed to the PHP page via Ajax in jQuery (through an onclick event on a website)....

Title search in SQL With replacement of noice words [on hold]


sql,sql-server,sql-server-2008
I have Two tables first us IMG_detail S.NO Title 1 women holding stack of gifts 2 Rear view of a man playing golf 3 Women holding gifts 4 Close-up of a golf ball on a tee 5 Businessman reading a newspaper and smiling and Second is tbl_NoiceWords SN Key 1...

Default the year based on month value


sql,sql-server
I have a query to display the year value from a table and default the current year. select distinct year(startdate) as syear, year(startdate) as eyear, 1 as sOrder from [TTS].[dbo].[Class] where year(startdate) <> year(getdate()) union all select year(getdate()) as syear, year(getdate()) as eyear, 0 as sOrder order by sOrder asc,...

MySQL: Select several rows based on several keys on a given column


mysql,sql,database
I have this mysql table structure: ------------------------------------ | item_id | meta_key | meta_value | ------------------------------------ 159 category Bungalow 159 location Lagos 159 price 45000 160 category Bungalow 160 location Abuja 160 price 53500 ... 350 category Bungalow 350 location Lagos 350 price 32000 What I'd like to do is select...

mysql_real_escape_string creates \ in server only not in local


php,sql
When I use mysql_real_escape_string in my localhost and I output the result of html in a table I have no problem. But when I use it on my server it outputs even the \ This is how I use it: $_GETVARS['txtEmpNum'] = mysql_real_escape_string($_GETVARS['txtEmpNum']); $_GETVARS['txtLName'] = mysql_real_escape_string($_GETVARS['txtLName']); $_GETVARS['txtFName'] = mysql_real_escape_string($_GETVARS['txtFName']); $varSQL...

How to call MySQL view in Struts2 or Hibernate


java,mysql,hibernate,java-ee,struts2
I have a view in MySQL database CREATE VIEW CustInfo AS SELECT a.custName, a.custMobile, b.profession, b.companyName, b.annualIncome FROM customer a INNER JOIN cust_proffessional_info b ON a.cust_id=b.cust_id Is there any way that i can call this view using Struts2 or in Hibernate. I have tried to search it but could not...

Having two arrays in variable php


php,mysql,arrays,variables,multidimensional-array
i've this code, i'm trying to explode my date e.g "04 June 2015" to have the day and the month in other 2 variables, but i have all in a method, with an array, is it possible, my code doesn't work, it writes "Array[0]" why? while ($i < $number) {...

The column name “FirstName” specified in the PIVOT operator conflicts with the existing column name in the PIVOT argument


sql,sql-server,sql-server-2008
I am getting the following error message when I am trying to do replace null to zero. The column name "jan" specified in the PIVOT operator conflicts with the existing column name in the PIVOT argument. Query below: select * from(select isnull(jan,0),isnull(feb,0),sum(data) as amount )as p pivot( sum(amount) for month...

If I export my database with phpmyadmin will it lock my tables or take my database down?


mysql,database,phpmyadmin
I want to create a backup of my database using the phpmyadmin export function. the database can't have any down time so I need to know if running a database export will take the database down? I've looked on-line but all I get is instructions on how to export. No...

Ignore Group if LIMIT is not reached in MySQL


mysql
I am working on a rather tricky SQL for my level of knowledge. I have searched and searched for an answer but haven't came across anything. Hopefully someone can shed some light on this. How can you stop SQL from outputting group of rows if the limit set is not...

Add 1 to datediff result if time pass 14:30 or 2:30 PM


sql,ms-access,ms-access-2007
Here is my SQL query which returns number of days by subtracting current date from specified date and returns exact as i need, but in addition i want to add 1 to result if current time passes 14:30 or 2:30. My query SELECT reservations.customerid, DateDiff("d",reservations.checkin_date,Now()) AS Due_nights FROM reservations Am...

Fatal error in if stament [duplicate]


php,mysql,mysqli
This question already has an answer here: Compile Error: Cannot use isset() on the result of an expression 2 answers Error - Cannot use isset() on the result of an expression (you can use "null !== expression" instead) I'm getting the error above the start of the if statement...

PHP: While loop not working after adjusting SELECT for SQL injection prevention


php,mysql,select,sql-injection,associative-array
I am trying to set up PHP queries for MySQL in a way to prevent SQL injection (standard website). I had a couple of INSERT queries where changing this worked well but on the following SELECT I keep getting an error since the update and it looks like the while...

Matplotlib: Plot the result of an SQL query


python,sql,matplotlib,plot
from sqlalchemy import create_engine import _mssql from matplotlib import pyplot as plt engine = create_engine('mssql+pymssql://**:****@127.0.0.1:1433/AffectV_Test') connection = engine.connect() result = connection.execute('SELECT Campaign_id, SUM(Count) AS Total_Count FROM Impressions GROUP BY Campaign_id') for row in result: print row connection.close() The above code generates an array: (54ca686d0189607081dbda85', 4174469) (551c21150189601fb08b6b64', 182) (552391ee0189601fb08b6b73', 237304) (5469f3ec0189606b1b25bcc0',...

How to search images by name inside a folder?


php,mysql,image
I have a MySQL table with a column "rounds" and each "rounds" has his own photos. Exemple round1 has photos from start=380 end=385. This means it has 6 photos and the name of the photos contains 380,381,382,383,384 or 385 inside. I use this PHP code to display the photos from...

How to create multiple jquery form fields and insert in mysql database, without using mysql_real_escape_string


javascript,php,jquery,mysql
I need to create dynamic text fields with a button (which simply adds a new row of dynamic text fields). Then I need to INSERT the data in a in the database. I borrowed an example of what I need done from someone else's question, the problem is, it uses...

Pull information from SQL database and getting login errors


php,sql,database
I am creating a very small, simple CRM for a company, they require the function to be able to view the last 25 orders via the dashboard. The orders are added via a Order-add form within the CRM. When adding the following code to the CRM I get an error:...

How do I display my mysql table column headers in my php/html output?


php,html,mysql,table,data
2 Questions... Scenario: I would like to query my database table via a form and then display the results that occur(if there are results) and my current situation is that it does work but it clears the form completely and leaves my to an empty page with just the results...

How to program a recurring billing/invoice system using PHP and MySQL


php,mysql
I have already programmed a basic invoicing system using PHP/MySQL which has a table structure as follows; Invoice table; invoice_id, invoice_date, customer_id, etc Invoice line table; invoice_line_id, invoice_id, quantity, price, description, etc I need the system to automatically generate future invoices at set intervals (for example every 1 week or...

SQL Customized search with special characters


sql,sql-server,sql-server-2008
I am creating a key-wording module where I want to search data using the comma separated words.And the search is categorized into comma , and minus -. I know a relational database engine is designed from the principle that a cell holds a single value and obeying to this rule...

Can someone explain to me how this statement is an exclude?


sql,sql-server,tsql
Can someone tell me how this is an exclude? Assuming that tableID is auto generated and in columnY there can be value of 0 or 1. This statement should exclude everything were columnY has a value of 1. SELECT * FROM [table].[dbo].[one] AS t1 LEFT JOIN [table].[dbo].[one] AS t2 ON...

C# - Can't connect to remote MySQL server


c#,mysql
My problem is that I can't connect to my website remote MySQL server. I have read all answers in stackoverflow.com, but I can't find right answer. Here's my C# code: using System; using System.Collections.Generic; using System.Data; using System.Data.SqlClient; namespace ConsoleApplication3 { class Program { static void Main(string[] args) { SqlConnection...

How to select next row after select in SQL Server?


sql,sql-server
I have this issue, I need your help: EmpNo Shift_Date Shift1 shift2 stamp_Date stamp_Time stamp_Type 426 2015-04-12 A 2015-04-12 10:09:00.000 I 426 2015-04-15 B C 2015-04-15 23:46:00.000 I 426 2015-04-15 B C 2015-04-15 23:45:00.000 O 426 2015-04-16 OF 2015-04-16 07:02:00.000 O 426 2015-04-17 A 2015-04-17 07:34:00.000 I 426 2015-04-18 A...

compare today's date with unix timestamp value in database


php,mysql
In database I am storing date value unix timestamp value for e.g.'1434952110' using time() now I am trying to compare todays value with this value as below $jobpostdate = date("Y-m-d");//today's date passing in database to compare query $sql = "SELECT jsp_title, jsp_subtitle, jsp_desc, jsp_location, jsp_date "; $sql .= "FROM js_projects...

Get unique row by single column where duplicates exist


sql,sql-server
I have the following DB table called messages with columns: thread_id, sender_id, receiver_id, date Example: +-----------+------------+-------------+-------------+ | thread_id | sender_id | receiver_id | date | +----------------------+-----------+-----------------+ | 1 | 1 | 2 | 11/06/2015 | | 1 | 2 | 1 | 14/06/2015 | | 1 | 1 | 2...

SQL: overcoming no ORDER BY in nested query


sql,sqlite
I have a table t as follows: CREATE TABLE t( id INTEGER PRIMARY KEY AUTOINCREMENT, date TEXT, a TEXT, b TEXT ); Sample Data 1|2015-06-15|a1 15|b1 15 2|2015-06-15|a2 15|b2 15 3|2015-06-16|a1 16|b1 16 4|2015-06-16|a2 16|b2 16 5|2015-06-17|a1 17|b1 17 6|2015-06-17|a2 17|b2 17 I want to select all values of a...

oracle sql error Case When Then Else


sql,oracle,oracle11g
SELECT * FROM FirstTable WHERE RowProcessed = 'N' AND ( CASE WHEN EXISTS(SELECT top 1 FROM SecondTable) THEN 1 ELSE EXISTS( SELECT SecondTable.RowProcessed FROM SecondTable WHERE FirstTable.Key = SecondTable.Key AND SecondTable.RowProcessed = 'Y' ) END ) AND OtherConditions Case When then else in where clause. Not sure about the syntax....

Foreign key in C#


c#,sql,sql-server,database
I have two tables, A and B, in a dataset in SQL Server; I have created a connection to the dataset in a c# project in visual studio. How can I create a foreign key ( A is the parent) between my two tables ? I want to create the...

Take thousand value in SQL


sql,sql-server
I have values like below I need to take only the thousand value in sql. 38,635.123 90,232.89 123,456.47888 I need to take result as below. 635 232 456...

Can Rails deal with DB uniqueness without index?


mysql,ruby-on-rails,rdbms
I see add_index ~ unique: true statement in schema.rband think uniqueness is constraint for table, not for index.Using index is one of way realizing uniqueness, Programmer should not designate to the RDBMS "how" and index quicken searching but take costs inserting. In fact, is there another way to keep uniqueness...

Php Mysql Query not working properly


php,mysql
I have a table name tblnetworkstatus and I have 11 columns Id issue_name affected_server affected_service issue_type priority duration status start_date end_date description I am getting id in affected_server and affected_service which I am storing in my DB, now I have three situations Either both affected_server and affected_service has been selected...