linux,multithreading,linux-kernel , How does the kernel separate threads from processes


How does the kernel separate threads from processes

Question:

Tag: linux,multithreading,linux-kernel

Suppose I have a browser process like Firefox, that has pid = 123. Firefox has 5 opened tabs each running in a separate thread, so in total it has 5 threads.

  1. So I want to know in depth, how the kernel will separate the process into the thread to execute in struct task_struct or in the thread_info.

  2. Like struct task_struct is a task descriptor of the task list. where does struct task_struct contain a reference or a link to these five threads.

  3. Does the struct thread_struct of a process like Firefox contain reference to all the 5 thread

    OR

    each thread is treated like a process inside the Linux kernel.


Answer:

Unlike Windows, Linux does not have an implementation of "threads" in the kernel. The kernel gives us what are sometimes called "lightweight processes", which are a generalization of the concepts of "processes" and "threads", and can be used to implement either.

It may be confusing when you read kernel code and see things like thread_struct on the one hand, and pid (process ID) on the other. In reality, both are one and the same. Don't be confused by the terminology.

Each lightweight process has a completely different thread_info and task_struct (with embedded thread_struct). You seem to think that the task_struct of one lightweight process should have pointers to the task_structs of other (userspace) "threads" in the same (userspace) "process". This is not the case. Inside the kernel, each "thread" is a separate process, and the scheduler deals with each one separately.

Linux has a system call called clone which is used to create new lightweight processes. When you call clone, you must provide various flags which indicate what will be shared between the new process and the existing process. They can share their address space, or they can each have a different address space. They can share their open files, or they can each have their own list of open files. They can share their signal handlers, or they can each have their own signal handlers. They can be in the same "thread group", or they can be in different thread groups. And so on...

Although "threads" and "processes" are the same thing in Linux, you can implement what we normally think of as "processes" by using clone to create processes which do not share their address space, open files, signal handlers, etc.

You can also implement what we normally think of as "threads" by using clone to create processes which DO share their address space, open files, signal handlers, etc.

If you look at the definition of task_struct, you will find that it has pointers to other structs such as mm_struct (address space), files_struct (open files), sighand_struct (signal handlers), and so on. When you clone a new "process", all of these structs will be copied. When you clone a new "thread", these structs will be shared between the new and old task_structs -- they will both point to the same mm_struct, the same files_struct, and so on. Either way, you are just providing different flags to clone to tell it what to copy, and what to share.

I just mentioned "thread groups" above, so you might wonder about that. In short, each "thread" in a "process" has its own PID, but they all share the same TGID (thread group ID). The TGIDs are all equal to the PID of the first program thread. Userspace "PIDs", like those shown in ps, or in /proc, are actually "TGIDs" in the kernel. Naturally, clone has a flag to determine whether a new lightweight process will have a new TGID (thus putting it in a new "thread group") or not.

UNIX processes also have "parents" and "children". There are pointers in a Linux task_struct which implement the parent-child relationships. And, as you might have guessed, clone has a flag to determine what the parent of a new lightweight process will be. It can either be the process which called clone, OR the parent of the process which called clone. Can you figure out which is used when creating a "process", and which is used when creating a "thread"?

Look at the manpage for clone; it will be very educational. Also try strace on a program which uses pthreads to see clone in use.

(A lot of this was written from memory; others should feel free to edit in corrections as necessary)


Related:


Extra backslash when storing grep in a value


linux,bash
In a bash script I have: Check="grep -e '"'\(-S mount\)'"' /etc/audit/audit.rules" set -x When you run it it shows it as: CHECK='grep -e '\''\(-S mount\)'\'' /etc/audit/audit.rules' Now it works exactly what I want but I want to understand it. Why is there 2 extra \'s?...

How to extract first letters of dashed separated words in a bash variable?


linux,string,bash,shell,variables
I would like to extract the first letter of dashed separated words value of my bash variable, like this: MY_TEXT=this-is-my-custom-text I would like to create a second variable like this: MY_INITIALS=timct...

ret_from_syscall source code and when it is called


linux,linux-kernel,kernel,linux-device-driver,system-calls
In the below call trace we see function called ret_from_syscall. Which function is this ? When it will called during system call ? Where is the corresponding code for this ? May 7 16:40:34.322086 warn TCU-0 kernel: [cf83ddc0] [00009751] 0x9751 (unreliable) May 7 16:40:34.322086 warn TCU-0 kernel: [cf83ddd0] [c00469ac] do_syslog+0x198/0x424...

How to append entry the end of a multi-line entry using any of stream editors like sed or awk


linux,bash,awk,sed,sh
I am trying to use a script to append the host name at the end of a multi-line entry of a specific Host_Alias field in sudoers file. The current sudoers file has an entry similar to : Host_Alias srv_linuxestate= \ host10,host12,host13,host1,host50,\ host16,host1,host2,host11,host15,host21,\ host3,host14 My required output would be something like...

Syncing Vagrant VMs across different physical servers


linux,vagrant,backup,virtual-machine,sync
I'm using Vagrant to deploy my VMs and my current setup looks like this: server1 = VM1, VM2, VM3 ( main production server ) server2 = VM1, VM2, VM3 ( backup server ) My questions is, can I somehow sync the VMs across the different physical servers in case one...

java multithreading start() and run() [duplicate]


java,multithreading
This question already has an answer here: Java: What's the difference between Thread start() and Runnable run() 10 answers Below is my Multithreading class: public class Multithreading extends Thread{ public void run(){ for(int i=1;i<5;i++){ try{ Thread.sleep(500); }catch(InterruptedException e){ System.out.println(e); } System.out.println(i); } } public static void main(String args[]) {...

sed and PHP tags


regex,linux,sed
I have a problem using SED. I have a php file whit this structure in the very first line: <?php echo 'first' ?><?php echo 'second' ?><?php echo 'third';?> I'm trying to remove the first two statements and have as a result: <?php echo 'third';?> I've tried this code: sed -i...

Django MySQLClient pip compile failure on Linux


python,linux,django,gcc,pip
Django documentation as of v1.8 recommends using mysqlclient connector for the framework. I'm attempting to pip install the package on Ubuntu 14.04 with Python 3.4 and running into a GCC error that I'm unable to find reference to. I'm not an expert on compiling software, so was hoping somebody can...

how to modify an array value with given index?


arrays,linux,bash
I want to modify an array cell, which I can do when I know the cell as a number. However here my cell position is given by $i. pomme[`${i}`]="" I tried without the `` and it doesn't work either? How am I suppose to do it?...

How can we make a thread to sleep for a infinite time in java?


java,multithreading,thread-sleep
Pls help me to understand how can we make a thread to sleep for a infinite time period .

Is there standard implementation for thread block/resume in java SE?


java,multithreading,wait
I need to block execution of a thread until resumed from another thread. So I wrote my own implementation using wait() method. Which seems to be working, but it is far from simple. Is there any ready to use solution? Preferably in java SE 6? Or do I have to...

Images not appearing on WPF form when loading asynchronously


c#,wpf,multithreading,listbox,backgroundworker
I'm attempting to display (in a ListBox with a custom DataTemplate) a series of BitmapSource frames (thumbnails) extracted from a multi-page tiff image. When I process the tiff on the UI thread, and either directly add the images to a listbox's item collection or add them to a bound ObservableCollection,...

performance issues executing list of stored procedures


c#,multithreading,performance,loops
I'm having some performance issues when starting my windows service, the first round my lstSps is long (about 130 stored procedures). Is there anyway to speed this up (except for speeding the stored procedures up)? When the foreach is over and goes over to the second round it goes faster,...

What are correct permissions for Linux Apache2 PHP 5.3 log file?


php,linux,apache,logging,permissions
I discovered the reason why I was not getting entries into my php_errors.log file related to permissions. Right now, I have set it to 666 (rw-rw-rw-) but surely this is a security weakness? Thus, my question. php.ini file: error_log /var/log/httpd/php_errors.log log_errors On # ls -ld /var/log /var/log/httpd /var/log/httpd/php_errors.log drwxr-xr-x 6...

Calling dispatch_sync from a concurrent queue - does it block entirely?


ios,objective-c,multithreading,swift,grand-central-dispatch
Let's say I hypothetically call a dispatch_sync from a concurrent queue - does it block the entire queue or just that thread of execution?

Delete some lines from text using Linux command


linux,shell,sed,grep,pattern-matching
I know how to match text using regex patterns but not how to manipulate them. I have used grep to match and extract lines from a text file, but I want to remove those lines from the text. How can I achieve this without having to write a python or...

How can I know the lock information in java?


java,multithreading,locking
Is there any tool or way that can get all the information about the locks in java? for example, if there is a java program, it creates two threads, and both threads require locks for some variable. Is there any tools that can output the information like which thread locks...

Linux - sh script - download multiple files from FTP


linux,ftp,sh
I need script that can download files from ftp via my sh code I have use expect with ftp, but if I do for loop inside code, I got wrong # args: should be "for start test next command" while executing "for v in "a b c"" My code /usr/bin/expect...

Linux-wget command


linux,shell,wget
I need a quick help on customizing my wget command in a shell script: The wget command looks something like this: wget http://infamvn:8081/nexus/content/groups/LDM_REPO_LIN64/com/infa/com.infa. products.ldm.ingestion.server.scala/10.0.0.135.527-SNAPSHOT/com.infa.products.ldm.ingestion.server.scala-10.0.0.135.527-20150622.210643-1-sources.jar Here I'd like to add the 10.0.0.135.527 in a variable, so I created a script something like this: n = 10.0.0.135.527 wget...

Multiple Threads searching on same folder at same time


c#,multithreading,file-search
Currently I have a .txt file of about 170,000 jpg file names and I read them all into a List (fileNames). I want to search ONE folder (this folder has sub-folders) to check if each file in fileNames exists in this folder and if it does, copy it to a...

AWK count number of times a term appear with respect to other columns


linux,shell,command-line,awk,sed
Given a CSV file: id, fruit, binary 1, apple, 1 2, orange, 0 3, pear, 1 4, apple, 0 5, peach, 0 6, apple, 1 How can i calculate for each unique values in fruit, the number of times the binary value =1 / number of occurences of that fruit...

linux running command as root from c code that run as normal user


c++,linux
I have a c++ code and I need to running from it a command to adjust the system time. so I thought using system("su root -c date hh:mm") command from my c++ code. The problem is that when I write 'su root -c date hh:mm' in the terminal its requires...

AWK write to new column base on if else of other column


linux,bash,shell,awk,sed
I have a CSV file with columns A,B,C,D. Column D contains values on a scale of 0 to 1. I want to use AWK to write to a new column E base in values in column D. For example: if value in column D <0.7, value in column E =...

Git post-receive hook is not executed


linux,git,githooks,git-post-receive
The following post-receive hook: #!/bin/bash echo "-> Post-receive test" is not executed when pushing to my remote repository. The remote is ssh://[email protected]:2222/home/git/repo.git (it's a VM) and works, as when I manually checkout it I see the modifications I've made. Some additional informations ... $ ls -al /home/git drwxr-xr-x 7 git...

How can I resolve the “Could not fix timestamps in …” “…Error: The requested feature is not implemented.”


linux,build,f#
I have been trying to build a project in F# on Linux that I have located here on github. It's a basic kata project that I am working on as a demo. However on Linux (specifically Ubuntu 14.04 LTS Desktop) I haven't been able to get it to build yet...

sed string with special character New


linux,sed,special-characters
when i use this script to replace : mrm.fr.mycompany.com by 10.70.89.40:8081/artifactory sed -i -e "s/mrm.fr.mycompany.com/10.70.89.40:8081/artifactory/g" config.xml i have the error : sed: -e expression n°1, caractère 41: option inconnue pour `s' can anyone help me thanks in advance regard, Youssef...

While loop in bash using variable from txt file


linux,bash,rhel
I am new to bash and writing a script to read variables that is stored on each line of a text file (there are thousands of these variables). So I tried to write a script that would read the lines and automatically output the solution to the screen and save...

Bash modify CSV to change a field


linux,bash,awk
I have a very big CSV file (aprox. 10.000 rows and 400 columns) and I need to modify certain columns (like 15, 156, 220) to change format from 20140321132233 to 2014-03-21 13:22:33. All fields that I need to modify are datetime. I saw some examples using awk but for math...

How to extract single-/multiline regex-matching items from an unpredictably formatted file and put each one in a single line into output file?


linux,shell,unix,replace,grep
I have a very huge file which looks like this: <a>text</a>text blah <b>data1</b>abc<b>data2</b> <b>data3</b>blahblah <c>text</c> <d>text</d> <x>blahblah<b>data4 data5 data6</b> <b>data7 </x> That is, its formatting is unpredictable. I need to extract each <b>...</b> item (it might contain multiline text!) and put every one of them in a single separate line....

Java how to limit number of threads acting on method


java,multithreading,memory-management
I have java method in my web application doing heavy file operation. The thing is, if more than 5 threads come simultaneously (which will come in testing phase) it breaks down. I mean it cannot handle heavy traffic. That's why I want to handle maximum 5 requests at a time...

Set Label From Thread


vb.net,multithreading,winforms
Form1.vb Imports System.Threading Public Class Form1 Dim demoThread As Thread Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click Dim Start As New Class1 Me.demoThread = New Thread( _ New ThreadStart(AddressOf Start.ThreadProcSafe)) Me.demoThread.Start() End Sub Delegate Sub SetTextCallback([text] As String) Public Sub SetText(ByVal [text] As String) ' InvokeRequired required...

wait for an event regulary


multithreading,events
In a program I need to wait for an event (keypress) and get it's char. after that program will continue. this progress will Repeat several time. in my first try, codes run and any character did not save. I find out I should use threading but I am not Familiar...

Using an ad-hoc libc with a tool which is an argument of another tool


linux,shared-libraries
I need to use a particular libc to run a tool (cp). The problem is that this tool has to be used as argument of another tool (for example timeout) and I don't want to use the modified libc with this one. I tried to do: timeout 10 LD_LIBRARY_PATH=/path/to/mod/libc/ cp...

make error during building webkitgtk


linux,makefile,cmake,make
I use UBuntu 14.04 LTS. I need to build webkitgtk 2.8.3 Here is an example instruction which I have used: linuxfromscratch When I run sudo make -j8 I get following log: Scanning dependencies of target JavaScriptCore-4-gir Scanning dependencies of target fake-generated-webkitdom-headers [ 0%] Scanning dependencies of target WebKit2-fake-api-headers Scanning dependencies...

Web API - Set each thread with the HttpRequestMessage id?


c#,.net,multithreading,task-parallel-library,web-api
I have a web api coded in c#. The web api uses functionality which is shared with other in-house components. it depends on single threaded flows and uses thread local storage to store objects, and session information. Please don't say if it's good or bad, that's what I have to...

Does wait() need synchronization on local variable


java,multithreading,synchronization
I had this code (which was working fine): public static void runOnUiThread(Activity c, final Runnable action) { // Check if we are on the UI Thread if (Looper.getMainLooper() == Looper.myLooper()) { // If we are, execute immediately action.run(); return; } // Else run the runnable on the UI Thread and...

Java read bytes from Socket on Linux


linux,windows,sockets,network-programming,raspberry-pi
I'm trying to send a file from my Windows machine to my Raspberry-Pi 2, and I have a client and a server. The client should be able to send a zip file over the network to my server on my linux machine. I know my client and server work on...

Java 5 Multi threading, catch thread exceptions


java,multithreading,exception,concurrency,java-5
I've got a class that generates threads (file i/o). I need to catch exceptions in the thread - I don't want to do anything fancy, I want to kill the main thread, rather, stop processing altogether so it can start over. If I catch the exceptions in the thread, that's...

NASM: copying a pointer from a register to a buffer in .data


linux,assembly,nasm,x86-64
I am new to asm. I am trying to copy a pointer from a register to a .data variable using NASM, on linux 64-bit. Concider this program: section .data ptr: dq 0 section .text global _start _start: mov [ptr], rsp mov rax, 60 mov rdi, 0 syscall Here I try...

Force linux to use php as php55


php,linux,fedora
Helllo, I have a little problem here. I have PHP 5.3 installed and it's accessible via php command, I also have php 5.5 accessible as php55 command. Now I need to force linux to use php55 when I write php command. Simple way: How I switch the versions of php?...

What does it indicate if /proc/PID/maps shows zero for all addresses?


linux,linux-kernel
I'm debugging a problem with a Linux DNS server. Curiously, when I look at /proc/PID/maps for the DNS server process, this is what I get: 00000000-00000000 r-xp 00000000 00:0e 2344 /usr/sbin/unbound 00000000-00000000 rw-p 00000000 00:0e 2344 /usr/sbin/unbound 00000000-00000000 ---p 00000000 00:00 0 00000000-00000000 rw-p 00000000 00:00 0 [heap] 00000000-00000000 rw-p...

Atomic/not-atomic mix, any guarantees?


c++,multithreading,c++11,atomic
Let's I have GUI thread with code like this: std::vector<int> vec; std::atomic<bool> data_ready{false}; std::thread th([&data_ready, &vec]() { //we get data vec.push_back(something); data_ready = true; }); draw_progress_dialog(); while (!data_ready) { process_not_user_events(); sleep_a_little(); } //is it here safe to use vec? As you see I not protect "vec" by any kind of...

Get system startup time (without reading /proc/uptime)


php,linux
I cannot open /proc/uptime due to open_basedir restriction. The command uptime is too old and doesn't have the -s flag support. How can I - in PHP - get time when the server started? My current code is this, but it does not work on the production server (for the...

Ignore first few lines and last few lines in a file Linux


linux,awk
I have a file like this and would like to print $0 except the first two and last three lines in linux. Tried awk command but no luck, is there any options I am using the following command - I suppose I am doing something wrong, but not able to...

std::condition_variable – notify once but wait thread wakened twice


c++,multithreading
Here's a simple C++ thread pool implementation. It's an altered version orginated from https://github.com/progschj/ThreadPool. #ifndef __THREAD_POOL_H__ #define __THREAD_POOL_H__ #include <vector> #include <queue> #include <memory> #include <thread> #include <chrono> #include <mutex> #include <condition_variable> #include <future> #include <functional> #include <stdexcept> namespace ThreadPool { class FixedThreadPool { public: FixedThreadPool(size_t); template<class F, class......

What happens if all node.js's worker threads are busy


javascript,node.js,multithreading
I try to understand how node.js works and although I have read this article: When is the thread pool used? I am not sure what happens if all worker threads are busy and another async I/O operation is ready to be executed. If I got this http://www.future-processing.pl/blog/on-problems-with-threads-in-node-js/ article right, the...

How does the kernel separate threads from processes


linux,multithreading,linux-kernel
Suppose I have a browser process like Firefox, that has pid = 123. Firefox has 5 opened tabs each running in a separate thread, so in total it has 5 threads. So I want to know in depth, how the kernel will separate the process into the thread to execute...

Looking to pause a thread using thread.sleep


java,multithreading
I am using a mouse listener for mouse pressed and released. When the mouse is pressed I want to have a counter incrementing a variable, and when the mouse is released I want to decrement that variable. Right now, my code is working and does that but the increment is...

fread(), solaris to unix portability and use of uninitialised values


c,linux,memory,stack,portability
Valgrind found the following error and I, after reading the documentation, the code and other questions in here couldn't figure it out why. Valgrind: first warning ~$ valgrind --vgdb=yes --vgdb-error=0 --read-var-info=yes --leak-check=yes --track-origins=yes debitadmin* debitadmin ==20720== Conditional jump or move depends on uninitialised value(s) ==20720== at 0x4013BC6: initialise (dbg.c:199) ==20720==...