Decrease of transfer rate when copying large amount of dataVery slow transfer between sata hard disksUSB data transfer from windows to linuxHow to transfer large amount of data from Mac to Ubuntu computer?Data transfer from XP to UbuntuSlow copy/move transfer rateHow to find the maximum data transfer rate possible from the RAM of my laptopFile copy to USB3 HDD slows down after 16GBExternal Hard drive gets slow while copying lots of dataI can't transfer large file from ubuntu hard drive to windows hd?Moving large amount of files (~ 100 000)Any reason, why a data transfer from one external USB-3 disk to another external USB-3 disk may be slow?
Why do Australian milk farmers need to protest supermarkets' milk price?
Good allowance savings plan?
Why don't MCU characters ever seem to have language issues?
If Invisibility ends because the original caster casts a non-concentration spell, does Invisibility also end on other targets of the original casting?
Need some help with my first LaTeX drawing…
Draw arrow on sides of triangle
What does it mean when multiple 々 marks follow a 、?
Time dilation for a moving electronic clock
Extension of Splitting Fields over An Arbitrary Field
What is the difference between "shut" and "close"?
Ban on all campaign finance?
Plywood subfloor won't screw down in a trailer home
What is the likely impact on flights of grounding an entire aircraft series?
Am I not good enough for you?
Why would a jet engine that runs at temps excess of 2000°C burn when it crashes?
Potentiometer like component
"However" used in a conditional clause?
Replacing Windows 7 security updates with anti-virus?
When were linguistics departments first established
Counter-example to the existence of left Bousfield localization of combinatorial model category
The three point beverage
Is going from continuous data to categorical always wrong?
Is "history" a male-biased word ("his+story")?
What exactly is the purpose of connection links straped between the rocket and the launch pad
Decrease of transfer rate when copying large amount of data
Very slow transfer between sata hard disksUSB data transfer from windows to linuxHow to transfer large amount of data from Mac to Ubuntu computer?Data transfer from XP to UbuntuSlow copy/move transfer rateHow to find the maximum data transfer rate possible from the RAM of my laptopFile copy to USB3 HDD slows down after 16GBExternal Hard drive gets slow while copying lots of dataI can't transfer large file from ubuntu hard drive to windows hd?Moving large amount of files (~ 100 000)Any reason, why a data transfer from one external USB-3 disk to another external USB-3 disk may be slow?
I am using a Ubuntu 16.04.3 LTS system (4.10.0-40-generic) with two HDD's and several partitions on each disk. When I copy data (<5GB) between the two disk I get a transfer rate around 70 MB/s. However, when I try to copy a large amount of data (>30GB) from one disk to another I notice several performance issues.
My question is whether or not this behaviour is normal and to be expected in Linux systems?
Can anyone explain this to me and advise me how to avoid this performance decrease?
Below I will describe my observations. In the example I copied a disk image file of 54GB from sda8 (325 GB partition) to sdb8 (1.6TB partition)
1) Transfer rate decreases and iowait increases
When I try to copy more than 50 GB I notice that gradually the transfer rate decreases. I am monitoring the performance using glances, atop, iotop and iostat. At 30GB progress the transfer rate has dropped to 58 MB/s, at 46 GB to 36MB/s, at 52GB to 12 MB/s. After that the transfer rate really starts to fluctuate and drops below 1MB/s.
At the same time I see that iowait is increasing from initially 0% up to 62% at the end.
During copying disk sd8 has a 'busy' percentage between 40% and 60%. Disk sdb is 100% busy all the time.
Not only the transfer rate drops but also my system becomes less responsive. I expect the iowait to be the cause of that.
Is this normal behaviour? How can the decrease in performance be avoided?
2) IOwait stays high after copying
When copying has ended I notice that iowait is still high and gradually starts to reduce to normal values. This takes a couple of minutes. I think that during that time data is still written to sdb at a rate around 1 or 2 MB/s. Using iotop it looks like the process "jdb2/sdb4-8" is causing this disk write. During the time that IOwait is decreasing, my system still suffers from bad responsiveness. Also is see that disk sda is not busy anymore, but disk sdb is still operating at 100% busy.
What is causing that my system has bad responsiveness for a couple of minutes after the copying action?
Can this be avoided?
3) Copying from network drive increases the effects
When I try to copy from my Synology NAS to my local disk (sdb8) the effects are even worse. First the network drive is mounted to my system and then copying is started. Initially also a transfer rate of 70MB/s is realized, but the transfer rate drops must faster. After a couple of GB the transfer rate has dropped far below 1 MB/s.
Copying was tried using drag and drop from Nautilus, command "cp", command rsync, FreeFileSync application, but all showed poor performance.
What could be the cause that the performance decrease effects are worse using network drives?
Additional information
During copying "iostat -dx 5" was used to monitor the disk performance. Around 5 GB of copying progress monitoring shows:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0,00 0,00 530,40 0,00 68064,80 0,00 256,65 1,62 3,06 3,06 0,00 1,63 86,72
sdb 0,00 18767,20 0,20 112,40 23,20 73169,60 1300,05 144,32 1345,39 308,00 1347,23 8,88 100,00
When copying has progressed to around 52 GB it shows:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0,00 0,00 64,60 0,00 8268,80 0,00 256,00 0,22 3,41 3,41 0,00 1,76 11,36
sdb 0,00 1054,40 0,20 10,60 6,40 6681,60 1238,52 148,56 9458,00 0,00 9636,45 92,59 100,00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0,00 0,00 50,20 0,00 6425,60 0,00 256,00 0,16 3,09 3,09 0,00 1,64 8,24
sdb 0,00 2905,80 0,40 17,00 8,80 10289,60 1183,72 141,86 10199,77 652,00 10424,42 57,47 100,00
I realize that these are multiple questions, but I suspect these are all related to the same cause and hope that someone can clarify this to me.
hard-drive performance transfer iowait
asked Jan 14 '18 at 23:46
user3074126user3074126
2612
bumped to the homepage by Community♦ 21 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
I am using a Ubuntu 16.04.3 LTS system (4.10.0-40-generic) with two HDD's and several partitions on each disk. When I copy data (<5GB) between the two disk I get a transfer rate around 70 MB/s. However, when I try to copy a large amount of data (>30GB) from one disk to another I notice several performance issues.
My question is whether or not this behaviour is normal and to be expected in Linux systems?
Can anyone explain this to me and advise me how to avoid this performance decrease?
Below I will describe my observations. In the example I copied a disk image file of 54GB from sda8 (325 GB partition) to sdb8 (1.6TB partition)
1) Transfer rate decreases and iowait increases
When I try to copy more than 50 GB I notice that gradually the transfer rate decreases. I am monitoring the performance using glances, atop, iotop and iostat. At 30GB progress the transfer rate has dropped to 58 MB/s, at 46 GB to 36MB/s, at 52GB to 12 MB/s. After that the transfer rate really starts to fluctuate and drops below 1MB/s.
At the same time I see that iowait is increasing from initially 0% up to 62% at the end.
During copying disk sd8 has a 'busy' percentage between 40% and 60%. Disk sdb is 100% busy all the time.
Not only the transfer rate drops but also my system becomes less responsive. I expect the iowait to be the cause of that.
Is this normal behaviour? How can the decrease in performance be avoided?
2) IOwait stays high after copying
When copying has ended I notice that iowait is still high and gradually starts to reduce to normal values. This takes a couple of minutes. I think that during that time data is still written to sdb at a rate around 1 or 2 MB/s. Using iotop it looks like the process "jdb2/sdb4-8" is causing this disk write. During the time that IOwait is decreasing, my system still suffers from bad responsiveness. Also is see that disk sda is not busy anymore, but disk sdb is still operating at 100% busy.
What is causing that my system has bad responsiveness for a couple of minutes after the copying action?
Can this be avoided?
3) Copying from network drive increases the effects
When I try to copy from my Synology NAS to my local disk (sdb8) the effects are even worse. First the network drive is mounted to my system and then copying is started. Initially also a transfer rate of 70MB/s is realized, but the transfer rate drops must faster. After a couple of GB the transfer rate has dropped far below 1 MB/s.
Copying was tried using drag and drop from Nautilus, command "cp", command rsync, FreeFileSync application, but all showed poor performance.
What could be the cause that the performance decrease effects are worse using network drives?
Additional information
During copying "iostat -dx 5" was used to monitor the disk performance. Around 5 GB of copying progress monitoring shows:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0,00 0,00 530,40 0,00 68064,80 0,00 256,65 1,62 3,06 3,06 0,00 1,63 86,72
sdb 0,00 18767,20 0,20 112,40 23,20 73169,60 1300,05 144,32 1345,39 308,00 1347,23 8,88 100,00
When copying has progressed to around 52 GB it shows:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0,00 0,00 64,60 0,00 8268,80 0,00 256,00 0,22 3,41 3,41 0,00 1,76 11,36
sdb 0,00 1054,40 0,20 10,60 6,40 6681,60 1238,52 148,56 9458,00 0,00 9636,45 92,59 100,00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0,00 0,00 50,20 0,00 6425,60 0,00 256,00 0,16 3,09 3,09 0,00 1,64 8,24
sdb 0,00 2905,80 0,40 17,00 8,80 10289,60 1183,72 141,86 10199,77 652,00 10424,42 57,47 100,00
I realize that these are multiple questions, but I suspect these are all related to the same cause and hope that someone can clarify this to me.
hard-drive performance transfer iowait
asked Jan 14 '18 at 23:46
user3074126user3074126
2612
bumped to the homepage by Community♦ 21 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
I was going to talk about how HDD transfer rate is higher at the outside tracks where data starts and lower on the inside tracks where data ends but it would not explain why it would slow so drastically to ~1MB/s
– thomasrutter
Jan 14 '18 at 23:50
Could be related to ext4s delayed allocation. However on its own that shouldn't be causing such a regression.
– jdwolf
Jan 15 '18 at 1:01
add a comment |
I am using a Ubuntu 16.04.3 LTS system (4.10.0-40-generic) with two HDD's and several partitions on each disk. When I copy data (<5GB) between the two disk I get a transfer rate around 70 MB/s. However, when I try to copy a large amount of data (>30GB) from one disk to another I notice several performance issues.
My question is whether or not this behaviour is normal and to be expected in Linux systems?
Can anyone explain this to me and advise me how to avoid this performance decrease?
Below I will describe my observations. In the example I copied a disk image file of 54GB from sda8 (325 GB partition) to sdb8 (1.6TB partition)
1) Transfer rate decreases and iowait increases
When I try to copy more than 50 GB I notice that gradually the transfer rate decreases. I am monitoring the performance using glances, atop, iotop and iostat. At 30GB progress the transfer rate has dropped to 58 MB/s, at 46 GB to 36MB/s, at 52GB to 12 MB/s. After that the transfer rate really starts to fluctuate and drops below 1MB/s.
At the same time I see that iowait is increasing from initially 0% up to 62% at the end.
During copying disk sd8 has a 'busy' percentage between 40% and 60%. Disk sdb is 100% busy all the time.
Not only the transfer rate drops but also my system becomes less responsive. I expect the iowait to be the cause of that.
Is this normal behaviour? How can the decrease in performance be avoided?
2) IOwait stays high after copying
When copying has ended I notice that iowait is still high and gradually starts to reduce to normal values. This takes a couple of minutes. I think that during that time data is still written to sdb at a rate around 1 or 2 MB/s. Using iotop it looks like the process "jdb2/sdb4-8" is causing this disk write. During the time that IOwait is decreasing, my system still suffers from bad responsiveness. Also is see that disk sda is not busy anymore, but disk sdb is still operating at 100% busy.
What is causing that my system has bad responsiveness for a couple of minutes after the copying action?
Can this be avoided?
3) Copying from network drive increases the effects
When I try to copy from my Synology NAS to my local disk (sdb8) the effects are even worse. First the network drive is mounted to my system and then copying is started. Initially also a transfer rate of 70MB/s is realized, but the transfer rate drops must faster. After a couple of GB the transfer rate has dropped far below 1 MB/s.
Copying was tried using drag and drop from Nautilus, command "cp", command rsync, FreeFileSync application, but all showed poor performance.
What could be the cause that the performance decrease effects are worse using network drives?
Additional information
During copying "iostat -dx 5" was used to monitor the disk performance. Around 5 GB of copying progress monitoring shows:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0,00 0,00 530,40 0,00 68064,80 0,00 256,65 1,62 3,06 3,06 0,00 1,63 86,72
sdb 0,00 18767,20 0,20 112,40 23,20 73169,60 1300,05 144,32 1345,39 308,00 1347,23 8,88 100,00
When copying has progressed to around 52 GB it shows:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0,00 0,00 64,60 0,00 8268,80 0,00 256,00 0,22 3,41 3,41 0,00 1,76 11,36
sdb 0,00 1054,40 0,20 10,60 6,40 6681,60 1238,52 148,56 9458,00 0,00 9636,45 92,59 100,00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0,00 0,00 50,20 0,00 6425,60 0,00 256,00 0,16 3,09 3,09 0,00 1,64 8,24
sdb 0,00 2905,80 0,40 17,00 8,80 10289,60 1183,72 141,86 10199,77 652,00 10424,42 57,47 100,00
I realize that these are multiple questions, but I suspect these are all related to the same cause and hope that someone can clarify this to me.
hard-drive performance transfer iowait
asked Jan 14 '18 at 23:46
user3074126user3074126
2612
I am using a Ubuntu 16.04.3 LTS system (4.10.0-40-generic) with two HDD's and several partitions on each disk. When I copy data (<5GB) between the two disk I get a transfer rate around 70 MB/s. However, when I try to copy a large amount of data (>30GB) from one disk to another I notice several performance issues.
My question is whether or not this behaviour is normal and to be expected in Linux systems?
Can anyone explain this to me and advise me how to avoid this performance decrease?
Below I will describe my observations. In the example I copied a disk image file of 54GB from sda8 (325 GB partition) to sdb8 (1.6TB partition)
1) Transfer rate decreases and iowait increases
When I try to copy more than 50 GB I notice that gradually the transfer rate decreases. I am monitoring the performance using glances, atop, iotop and iostat. At 30GB progress the transfer rate has dropped to 58 MB/s, at 46 GB to 36MB/s, at 52GB to 12 MB/s. After that the transfer rate really starts to fluctuate and drops below 1MB/s.
At the same time I see that iowait is increasing from initially 0% up to 62% at the end.
During copying disk sd8 has a 'busy' percentage between 40% and 60%. Disk sdb is 100% busy all the time.
Not only the transfer rate drops but also my system becomes less responsive. I expect the iowait to be the cause of that.
Is this normal behaviour? How can the decrease in performance be avoided?
2) IOwait stays high after copying
When copying has ended I notice that iowait is still high and gradually starts to reduce to normal values. This takes a couple of minutes. I think that during that time data is still written to sdb at a rate around 1 or 2 MB/s. Using iotop it looks like the process "jdb2/sdb4-8" is causing this disk write. During the time that IOwait is decreasing, my system still suffers from bad responsiveness. Also is see that disk sda is not busy anymore, but disk sdb is still operating at 100% busy.
What is causing that my system has bad responsiveness for a couple of minutes after the copying action?
Can this be avoided?
3) Copying from network drive increases the effects
When I try to copy from my Synology NAS to my local disk (sdb8) the effects are even worse. First the network drive is mounted to my system and then copying is started. Initially also a transfer rate of 70MB/s is realized, but the transfer rate drops must faster. After a couple of GB the transfer rate has dropped far below 1 MB/s.
Copying was tried using drag and drop from Nautilus, command "cp", command rsync, FreeFileSync application, but all showed poor performance.
What could be the cause that the performance decrease effects are worse using network drives?
Additional information
During copying "iostat -dx 5" was used to monitor the disk performance. Around 5 GB of copying progress monitoring shows:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0,00 0,00 530,40 0,00 68064,80 0,00 256,65 1,62 3,06 3,06 0,00 1,63 86,72
sdb 0,00 18767,20 0,20 112,40 23,20 73169,60 1300,05 144,32 1345,39 308,00 1347,23 8,88 100,00
When copying has progressed to around 52 GB it shows:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0,00 0,00 64,60 0,00 8268,80 0,00 256,00 0,22 3,41 3,41 0,00 1,76 11,36
sdb 0,00 1054,40 0,20 10,60 6,40 6681,60 1238,52 148,56 9458,00 0,00 9636,45 92,59 100,00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0,00 0,00 50,20 0,00 6425,60 0,00 256,00 0,16 3,09 3,09 0,00 1,64 8,24
sdb 0,00 2905,80 0,40 17,00 8,80 10289,60 1183,72 141,86 10199,77 652,00 10424,42 57,47 100,00
I realize that these are multiple questions, but I suspect these are all related to the same cause and hope that someone can clarify this to me.
hard-drive performance transfer iowait
hard-drive performance transfer iowait
asked Jan 14 '18 at 23:46
user3074126user3074126
2612
asked Jan 14 '18 at 23:46
user3074126user3074126
2612
asked Jan 14 '18 at 23:46
user3074126user3074126
2612
asked Jan 14 '18 at 23:46
user3074126user3074126
2612
asked Jan 14 '18 at 23:46
user3074126user3074126
2612
2612
bumped to the homepage by Community♦ 21 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 21 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
I was going to talk about how HDD transfer rate is higher at the outside tracks where data starts and lower on the inside tracks where data ends but it would not explain why it would slow so drastically to ~1MB/s
– thomasrutter
Jan 14 '18 at 23:50
Could be related to ext4s delayed allocation. However on its own that shouldn't be causing such a regression.
– jdwolf
Jan 15 '18 at 1:01
add a comment |
I was going to talk about how HDD transfer rate is higher at the outside tracks where data starts and lower on the inside tracks where data ends but it would not explain why it would slow so drastically to ~1MB/s
– thomasrutter
Jan 14 '18 at 23:50
Could be related to ext4s delayed allocation. However on its own that shouldn't be causing such a regression.
– jdwolf
Jan 15 '18 at 1:01
I was going to talk about how HDD transfer rate is higher at the outside tracks where data starts and lower on the inside tracks where data ends but it would not explain why it would slow so drastically to ~1MB/s
– thomasrutter
Jan 14 '18 at 23:50
I was going to talk about how HDD transfer rate is higher at the outside tracks where data starts and lower on the inside tracks where data ends but it would not explain why it would slow so drastically to ~1MB/s
– thomasrutter
Jan 14 '18 at 23:50
Could be related to ext4s delayed allocation. However on its own that shouldn't be causing such a regression.
– jdwolf
Jan 15 '18 at 1:01
Could be related to ext4s delayed allocation. However on its own that shouldn't be causing such a regression.
– jdwolf
Jan 15 '18 at 1:01
add a comment |
1 Answer
1
active
oldest
votes
Unfortunately, this is normal and expected for your big file use case. Your case of two hard disks, and a 50G+ file eliminates a lot of misleading talk of "slow devices", "slow buses", and "slow filesystems", and you are left with the unexplained problem of a slow copy. You must have quite a bit of memory to get the performance you have for 30G files. System buffers are used, filled up, and after your copy command finishes, eventually will get flushed to the target, making real timing/rates somewhat difficult (even the "time" command will finish long before the buffers finally get flushed.
The only "workaround" I have found is to use a "copy" command which allows you to set up explicit buffers yourself, like tar or cpio can do. Setting a 2M buffer on tar allowed me to speed up a 10M/sec copy of a 50G file to about 35M/sec -- still much slower than the nominal 100M/sec I get on smaller files (or in Windows).
answered Jan 15 '18 at 5:20
ubfan1ubfan1
9,74941730
1
Thanx for your reply. In the meantime I have tried some additional test, and what I observe is in line with your answer.
– user3074126
Jan 16 '18 at 14:22
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "89"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2faskubuntu.com%2fquestions%2f995946%2fdecrease-of-transfer-rate-when-copying-large-amount-of-data%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Unfortunately, this is normal and expected for your big file use case. Your case of two hard disks, and a 50G+ file eliminates a lot of misleading talk of "slow devices", "slow buses", and "slow filesystems", and you are left with the unexplained problem of a slow copy. You must have quite a bit of memory to get the performance you have for 30G files. System buffers are used, filled up, and after your copy command finishes, eventually will get flushed to the target, making real timing/rates somewhat difficult (even the "time" command will finish long before the buffers finally get flushed.
The only "workaround" I have found is to use a "copy" command which allows you to set up explicit buffers yourself, like tar or cpio can do. Setting a 2M buffer on tar allowed me to speed up a 10M/sec copy of a 50G file to about 35M/sec -- still much slower than the nominal 100M/sec I get on smaller files (or in Windows).
answered Jan 15 '18 at 5:20
ubfan1ubfan1
9,74941730
1
Thanx for your reply. In the meantime I have tried some additional test, and what I observe is in line with your answer.
– user3074126
Jan 16 '18 at 14:22
add a comment |
Unfortunately, this is normal and expected for your big file use case. Your case of two hard disks, and a 50G+ file eliminates a lot of misleading talk of "slow devices", "slow buses", and "slow filesystems", and you are left with the unexplained problem of a slow copy. You must have quite a bit of memory to get the performance you have for 30G files. System buffers are used, filled up, and after your copy command finishes, eventually will get flushed to the target, making real timing/rates somewhat difficult (even the "time" command will finish long before the buffers finally get flushed.
The only "workaround" I have found is to use a "copy" command which allows you to set up explicit buffers yourself, like tar or cpio can do. Setting a 2M buffer on tar allowed me to speed up a 10M/sec copy of a 50G file to about 35M/sec -- still much slower than the nominal 100M/sec I get on smaller files (or in Windows).
answered Jan 15 '18 at 5:20
ubfan1ubfan1
9,74941730
1
Thanx for your reply. In the meantime I have tried some additional test, and what I observe is in line with your answer.
– user3074126
Jan 16 '18 at 14:22
add a comment |
Unfortunately, this is normal and expected for your big file use case. Your case of two hard disks, and a 50G+ file eliminates a lot of misleading talk of "slow devices", "slow buses", and "slow filesystems", and you are left with the unexplained problem of a slow copy. You must have quite a bit of memory to get the performance you have for 30G files. System buffers are used, filled up, and after your copy command finishes, eventually will get flushed to the target, making real timing/rates somewhat difficult (even the "time" command will finish long before the buffers finally get flushed.
The only "workaround" I have found is to use a "copy" command which allows you to set up explicit buffers yourself, like tar or cpio can do. Setting a 2M buffer on tar allowed me to speed up a 10M/sec copy of a 50G file to about 35M/sec -- still much slower than the nominal 100M/sec I get on smaller files (or in Windows).
answered Jan 15 '18 at 5:20
ubfan1ubfan1
9,74941730
Unfortunately, this is normal and expected for your big file use case. Your case of two hard disks, and a 50G+ file eliminates a lot of misleading talk of "slow devices", "slow buses", and "slow filesystems", and you are left with the unexplained problem of a slow copy. You must have quite a bit of memory to get the performance you have for 30G files. System buffers are used, filled up, and after your copy command finishes, eventually will get flushed to the target, making real timing/rates somewhat difficult (even the "time" command will finish long before the buffers finally get flushed.
The only "workaround" I have found is to use a "copy" command which allows you to set up explicit buffers yourself, like tar or cpio can do. Setting a 2M buffer on tar allowed me to speed up a 10M/sec copy of a 50G file to about 35M/sec -- still much slower than the nominal 100M/sec I get on smaller files (or in Windows).
answered Jan 15 '18 at 5:20
ubfan1ubfan1
9,74941730
answered Jan 15 '18 at 5:20
ubfan1ubfan1
9,74941730
answered Jan 15 '18 at 5:20
ubfan1ubfan1
9,74941730
answered Jan 15 '18 at 5:20
ubfan1ubfan1
9,74941730
9,74941730
1
Thanx for your reply. In the meantime I have tried some additional test, and what I observe is in line with your answer.
– user3074126
Jan 16 '18 at 14:22
add a comment |
1
Thanx for your reply. In the meantime I have tried some additional test, and what I observe is in line with your answer.
– user3074126
Jan 16 '18 at 14:22
1
1
Thanx for your reply. In the meantime I have tried some additional test, and what I observe is in line with your answer.
– user3074126
Jan 16 '18 at 14:22
Thanx for your reply. In the meantime I have tried some additional test, and what I observe is in line with your answer.
– user3074126
Jan 16 '18 at 14:22
add a comment |
Thanks for contributing an answer to Ask Ubuntu!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2faskubuntu.com%2fquestions%2f995946%2fdecrease-of-transfer-rate-when-copying-large-amount-of-data%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
I was going to talk about how HDD transfer rate is higher at the outside tracks where data starts and lower on the inside tracks where data ends but it would not explain why it would slow so drastically to ~1MB/s
– thomasrutter
Jan 14 '18 at 23:50
Could be related to ext4s delayed allocation. However on its own that shouldn't be causing such a regression.
– jdwolf
Jan 15 '18 at 1:01