Decrease of transfer rate when copying large amount of dataVery slow transfer between sata hard disksUSB data transfer from windows to linuxHow to transfer large amount of data from Mac to Ubuntu computer?Data transfer from XP to UbuntuSlow copy/move transfer rateHow to find the maximum data transfer rate possible from the RAM of my laptopFile copy to USB3 HDD slows down after 16GBExternal Hard drive gets slow while copying lots of dataI can't transfer large file from ubuntu hard drive to windows hd?Moving large amount of files (~ 100 000)Any reason, why a data transfer from one external USB-3 disk to another external USB-3 disk may be slow?

Why do Australian milk farmers need to protest supermarkets' milk price?

Good allowance savings plan?

Why don't MCU characters ever seem to have language issues?

If Invisibility ends because the original caster casts a non-concentration spell, does Invisibility also end on other targets of the original casting?

Need some help with my first LaTeX drawing…

Draw arrow on sides of triangle

What does it mean when multiple 々 marks follow a 、?

Time dilation for a moving electronic clock

Extension of Splitting Fields over An Arbitrary Field

What is the difference between "shut" and "close"?

Ban on all campaign finance?

Plywood subfloor won't screw down in a trailer home

What is the likely impact on flights of grounding an entire aircraft series?

Am I not good enough for you?

Why would a jet engine that runs at temps excess of 2000°C burn when it crashes?

Potentiometer like component

"However" used in a conditional clause?

Replacing Windows 7 security updates with anti-virus?

When were linguistics departments first established

Counter-example to the existence of left Bousfield localization of combinatorial model category

The three point beverage

Is going from continuous data to categorical always wrong?

Is "history" a male-biased word ("his+story")?

What exactly is the purpose of connection links straped between the rocket and the launch pad



Decrease of transfer rate when copying large amount of data


Very slow transfer between sata hard disksUSB data transfer from windows to linuxHow to transfer large amount of data from Mac to Ubuntu computer?Data transfer from XP to UbuntuSlow copy/move transfer rateHow to find the maximum data transfer rate possible from the RAM of my laptopFile copy to USB3 HDD slows down after 16GBExternal Hard drive gets slow while copying lots of dataI can't transfer large file from ubuntu hard drive to windows hd?Moving large amount of files (~ 100 000)Any reason, why a data transfer from one external USB-3 disk to another external USB-3 disk may be slow?













5















I am using a Ubuntu 16.04.3 LTS system (4.10.0-40-generic) with two HDD's and several partitions on each disk. When I copy data (<5GB) between the two disk I get a transfer rate around 70 MB/s. However, when I try to copy a large amount of data (>30GB) from one disk to another I notice several performance issues.



My question is whether or not this behaviour is normal and to be expected in Linux systems?

Can anyone explain this to me and advise me how to avoid this performance decrease?



Below I will describe my observations. In the example I copied a disk image file of 54GB from sda8 (325 GB partition) to sdb8 (1.6TB partition)



1) Transfer rate decreases and iowait increases

When I try to copy more than 50 GB I notice that gradually the transfer rate decreases. I am monitoring the performance using glances, atop, iotop and iostat. At 30GB progress the transfer rate has dropped to 58 MB/s, at 46 GB to 36MB/s, at 52GB to 12 MB/s. After that the transfer rate really starts to fluctuate and drops below 1MB/s.
At the same time I see that iowait is increasing from initially 0% up to 62% at the end.
During copying disk sd8 has a 'busy' percentage between 40% and 60%. Disk sdb is 100% busy all the time.
Not only the transfer rate drops but also my system becomes less responsive. I expect the iowait to be the cause of that.

Is this normal behaviour? How can the decrease in performance be avoided?



2) IOwait stays high after copying

When copying has ended I notice that iowait is still high and gradually starts to reduce to normal values. This takes a couple of minutes. I think that during that time data is still written to sdb at a rate around 1 or 2 MB/s. Using iotop it looks like the process "jdb2/sdb4-8" is causing this disk write. During the time that IOwait is decreasing, my system still suffers from bad responsiveness. Also is see that disk sda is not busy anymore, but disk sdb is still operating at 100% busy.

What is causing that my system has bad responsiveness for a couple of minutes after the copying action?

Can this be avoided?



3) Copying from network drive increases the effects

When I try to copy from my Synology NAS to my local disk (sdb8) the effects are even worse. First the network drive is mounted to my system and then copying is started. Initially also a transfer rate of 70MB/s is realized, but the transfer rate drops must faster. After a couple of GB the transfer rate has dropped far below 1 MB/s.
Copying was tried using drag and drop from Nautilus, command "cp", command rsync, FreeFileSync application, but all showed poor performance.

What could be the cause that the performance decrease effects are worse using network drives?



Additional information

During copying "iostat -dx 5" was used to monitor the disk performance. Around 5 GB of copying progress monitoring shows:



Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0,00 0,00 530,40 0,00 68064,80 0,00 256,65 1,62 3,06 3,06 0,00 1,63 86,72
sdb 0,00 18767,20 0,20 112,40 23,20 73169,60 1300,05 144,32 1345,39 308,00 1347,23 8,88 100,00


When copying has progressed to around 52 GB it shows:



Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0,00 0,00 64,60 0,00 8268,80 0,00 256,00 0,22 3,41 3,41 0,00 1,76 11,36
sdb 0,00 1054,40 0,20 10,60 6,40 6681,60 1238,52 148,56 9458,00 0,00 9636,45 92,59 100,00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0,00 0,00 50,20 0,00 6425,60 0,00 256,00 0,16 3,09 3,09 0,00 1,64 8,24
sdb 0,00 2905,80 0,40 17,00 8,80 10289,60 1183,72 141,86 10199,77 652,00 10424,42 57,47 100,00


I realize that these are multiple questions, but I suspect these are all related to the same cause and hope that someone can clarify this to me.










share|improve this question














bumped to the homepage by Community 21 mins ago


This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.















  • I was going to talk about how HDD transfer rate is higher at the outside tracks where data starts and lower on the inside tracks where data ends but it would not explain why it would slow so drastically to ~1MB/s

    – thomasrutter
    Jan 14 '18 at 23:50











  • Could be related to ext4s delayed allocation. However on its own that shouldn't be causing such a regression.

    – jdwolf
    Jan 15 '18 at 1:01















5















I am using a Ubuntu 16.04.3 LTS system (4.10.0-40-generic) with two HDD's and several partitions on each disk. When I copy data (<5GB) between the two disk I get a transfer rate around 70 MB/s. However, when I try to copy a large amount of data (>30GB) from one disk to another I notice several performance issues.



My question is whether or not this behaviour is normal and to be expected in Linux systems?

Can anyone explain this to me and advise me how to avoid this performance decrease?



Below I will describe my observations. In the example I copied a disk image file of 54GB from sda8 (325 GB partition) to sdb8 (1.6TB partition)



1) Transfer rate decreases and iowait increases

When I try to copy more than 50 GB I notice that gradually the transfer rate decreases. I am monitoring the performance using glances, atop, iotop and iostat. At 30GB progress the transfer rate has dropped to 58 MB/s, at 46 GB to 36MB/s, at 52GB to 12 MB/s. After that the transfer rate really starts to fluctuate and drops below 1MB/s.
At the same time I see that iowait is increasing from initially 0% up to 62% at the end.
During copying disk sd8 has a 'busy' percentage between 40% and 60%. Disk sdb is 100% busy all the time.
Not only the transfer rate drops but also my system becomes less responsive. I expect the iowait to be the cause of that.

Is this normal behaviour? How can the decrease in performance be avoided?



2) IOwait stays high after copying

When copying has ended I notice that iowait is still high and gradually starts to reduce to normal values. This takes a couple of minutes. I think that during that time data is still written to sdb at a rate around 1 or 2 MB/s. Using iotop it looks like the process "jdb2/sdb4-8" is causing this disk write. During the time that IOwait is decreasing, my system still suffers from bad responsiveness. Also is see that disk sda is not busy anymore, but disk sdb is still operating at 100% busy.

What is causing that my system has bad responsiveness for a couple of minutes after the copying action?

Can this be avoided?



3) Copying from network drive increases the effects

When I try to copy from my Synology NAS to my local disk (sdb8) the effects are even worse. First the network drive is mounted to my system and then copying is started. Initially also a transfer rate of 70MB/s is realized, but the transfer rate drops must faster. After a couple of GB the transfer rate has dropped far below 1 MB/s.
Copying was tried using drag and drop from Nautilus, command "cp", command rsync, FreeFileSync application, but all showed poor performance.

What could be the cause that the performance decrease effects are worse using network drives?



Additional information

During copying "iostat -dx 5" was used to monitor the disk performance. Around 5 GB of copying progress monitoring shows:



Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0,00 0,00 530,40 0,00 68064,80 0,00 256,65 1,62 3,06 3,06 0,00 1,63 86,72
sdb 0,00 18767,20 0,20 112,40 23,20 73169,60 1300,05 144,32 1345,39 308,00 1347,23 8,88 100,00


When copying has progressed to around 52 GB it shows:



Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0,00 0,00 64,60 0,00 8268,80 0,00 256,00 0,22 3,41 3,41 0,00 1,76 11,36
sdb 0,00 1054,40 0,20 10,60 6,40 6681,60 1238,52 148,56 9458,00 0,00 9636,45 92,59 100,00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0,00 0,00 50,20 0,00 6425,60 0,00 256,00 0,16 3,09 3,09 0,00 1,64 8,24
sdb 0,00 2905,80 0,40 17,00 8,80 10289,60 1183,72 141,86 10199,77 652,00 10424,42 57,47 100,00


I realize that these are multiple questions, but I suspect these are all related to the same cause and hope that someone can clarify this to me.










share|improve this question














bumped to the homepage by Community 21 mins ago


This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.















  • I was going to talk about how HDD transfer rate is higher at the outside tracks where data starts and lower on the inside tracks where data ends but it would not explain why it would slow so drastically to ~1MB/s

    – thomasrutter
    Jan 14 '18 at 23:50











  • Could be related to ext4s delayed allocation. However on its own that shouldn't be causing such a regression.

    – jdwolf
    Jan 15 '18 at 1:01













5












5








5


1






I am using a Ubuntu 16.04.3 LTS system (4.10.0-40-generic) with two HDD's and several partitions on each disk. When I copy data (<5GB) between the two disk I get a transfer rate around 70 MB/s. However, when I try to copy a large amount of data (>30GB) from one disk to another I notice several performance issues.



My question is whether or not this behaviour is normal and to be expected in Linux systems?

Can anyone explain this to me and advise me how to avoid this performance decrease?



Below I will describe my observations. In the example I copied a disk image file of 54GB from sda8 (325 GB partition) to sdb8 (1.6TB partition)



1) Transfer rate decreases and iowait increases

When I try to copy more than 50 GB I notice that gradually the transfer rate decreases. I am monitoring the performance using glances, atop, iotop and iostat. At 30GB progress the transfer rate has dropped to 58 MB/s, at 46 GB to 36MB/s, at 52GB to 12 MB/s. After that the transfer rate really starts to fluctuate and drops below 1MB/s.
At the same time I see that iowait is increasing from initially 0% up to 62% at the end.
During copying disk sd8 has a 'busy' percentage between 40% and 60%. Disk sdb is 100% busy all the time.
Not only the transfer rate drops but also my system becomes less responsive. I expect the iowait to be the cause of that.

Is this normal behaviour? How can the decrease in performance be avoided?



2) IOwait stays high after copying

When copying has ended I notice that iowait is still high and gradually starts to reduce to normal values. This takes a couple of minutes. I think that during that time data is still written to sdb at a rate around 1 or 2 MB/s. Using iotop it looks like the process "jdb2/sdb4-8" is causing this disk write. During the time that IOwait is decreasing, my system still suffers from bad responsiveness. Also is see that disk sda is not busy anymore, but disk sdb is still operating at 100% busy.

What is causing that my system has bad responsiveness for a couple of minutes after the copying action?

Can this be avoided?



3) Copying from network drive increases the effects

When I try to copy from my Synology NAS to my local disk (sdb8) the effects are even worse. First the network drive is mounted to my system and then copying is started. Initially also a transfer rate of 70MB/s is realized, but the transfer rate drops must faster. After a couple of GB the transfer rate has dropped far below 1 MB/s.
Copying was tried using drag and drop from Nautilus, command "cp", command rsync, FreeFileSync application, but all showed poor performance.

What could be the cause that the performance decrease effects are worse using network drives?



Additional information

During copying "iostat -dx 5" was used to monitor the disk performance. Around 5 GB of copying progress monitoring shows:



Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0,00 0,00 530,40 0,00 68064,80 0,00 256,65 1,62 3,06 3,06 0,00 1,63 86,72
sdb 0,00 18767,20 0,20 112,40 23,20 73169,60 1300,05 144,32 1345,39 308,00 1347,23 8,88 100,00


When copying has progressed to around 52 GB it shows:



Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0,00 0,00 64,60 0,00 8268,80 0,00 256,00 0,22 3,41 3,41 0,00 1,76 11,36
sdb 0,00 1054,40 0,20 10,60 6,40 6681,60 1238,52 148,56 9458,00 0,00 9636,45 92,59 100,00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0,00 0,00 50,20 0,00 6425,60 0,00 256,00 0,16 3,09 3,09 0,00 1,64 8,24
sdb 0,00 2905,80 0,40 17,00 8,80 10289,60 1183,72 141,86 10199,77 652,00 10424,42 57,47 100,00


I realize that these are multiple questions, but I suspect these are all related to the same cause and hope that someone can clarify this to me.










share|improve this question














I am using a Ubuntu 16.04.3 LTS system (4.10.0-40-generic) with two HDD's and several partitions on each disk. When I copy data (<5GB) between the two disk I get a transfer rate around 70 MB/s. However, when I try to copy a large amount of data (>30GB) from one disk to another I notice several performance issues.



My question is whether or not this behaviour is normal and to be expected in Linux systems?

Can anyone explain this to me and advise me how to avoid this performance decrease?



Below I will describe my observations. In the example I copied a disk image file of 54GB from sda8 (325 GB partition) to sdb8 (1.6TB partition)



1) Transfer rate decreases and iowait increases

When I try to copy more than 50 GB I notice that gradually the transfer rate decreases. I am monitoring the performance using glances, atop, iotop and iostat. At 30GB progress the transfer rate has dropped to 58 MB/s, at 46 GB to 36MB/s, at 52GB to 12 MB/s. After that the transfer rate really starts to fluctuate and drops below 1MB/s.
At the same time I see that iowait is increasing from initially 0% up to 62% at the end.
During copying disk sd8 has a 'busy' percentage between 40% and 60%. Disk sdb is 100% busy all the time.
Not only the transfer rate drops but also my system becomes less responsive. I expect the iowait to be the cause of that.

Is this normal behaviour? How can the decrease in performance be avoided?



2) IOwait stays high after copying

When copying has ended I notice that iowait is still high and gradually starts to reduce to normal values. This takes a couple of minutes. I think that during that time data is still written to sdb at a rate around 1 or 2 MB/s. Using iotop it looks like the process "jdb2/sdb4-8" is causing this disk write. During the time that IOwait is decreasing, my system still suffers from bad responsiveness. Also is see that disk sda is not busy anymore, but disk sdb is still operating at 100% busy.

What is causing that my system has bad responsiveness for a couple of minutes after the copying action?

Can this be avoided?



3) Copying from network drive increases the effects

When I try to copy from my Synology NAS to my local disk (sdb8) the effects are even worse. First the network drive is mounted to my system and then copying is started. Initially also a transfer rate of 70MB/s is realized, but the transfer rate drops must faster. After a couple of GB the transfer rate has dropped far below 1 MB/s.
Copying was tried using drag and drop from Nautilus, command "cp", command rsync, FreeFileSync application, but all showed poor performance.

What could be the cause that the performance decrease effects are worse using network drives?



Additional information

During copying "iostat -dx 5" was used to monitor the disk performance. Around 5 GB of copying progress monitoring shows:



Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0,00 0,00 530,40 0,00 68064,80 0,00 256,65 1,62 3,06 3,06 0,00 1,63 86,72
sdb 0,00 18767,20 0,20 112,40 23,20 73169,60 1300,05 144,32 1345,39 308,00 1347,23 8,88 100,00


When copying has progressed to around 52 GB it shows:



Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0,00 0,00 64,60 0,00 8268,80 0,00 256,00 0,22 3,41 3,41 0,00 1,76 11,36
sdb 0,00 1054,40 0,20 10,60 6,40 6681,60 1238,52 148,56 9458,00 0,00 9636,45 92,59 100,00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0,00 0,00 50,20 0,00 6425,60 0,00 256,00 0,16 3,09 3,09 0,00 1,64 8,24
sdb 0,00 2905,80 0,40 17,00 8,80 10289,60 1183,72 141,86 10199,77 652,00 10424,42 57,47 100,00


I realize that these are multiple questions, but I suspect these are all related to the same cause and hope that someone can clarify this to me.







hard-drive performance transfer iowait






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Jan 14 '18 at 23:46









user3074126user3074126

2612




2612





bumped to the homepage by Community 21 mins ago


This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.







bumped to the homepage by Community 21 mins ago


This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.














  • I was going to talk about how HDD transfer rate is higher at the outside tracks where data starts and lower on the inside tracks where data ends but it would not explain why it would slow so drastically to ~1MB/s

    – thomasrutter
    Jan 14 '18 at 23:50











  • Could be related to ext4s delayed allocation. However on its own that shouldn't be causing such a regression.

    – jdwolf
    Jan 15 '18 at 1:01

















  • I was going to talk about how HDD transfer rate is higher at the outside tracks where data starts and lower on the inside tracks where data ends but it would not explain why it would slow so drastically to ~1MB/s

    – thomasrutter
    Jan 14 '18 at 23:50











  • Could be related to ext4s delayed allocation. However on its own that shouldn't be causing such a regression.

    – jdwolf
    Jan 15 '18 at 1:01
















I was going to talk about how HDD transfer rate is higher at the outside tracks where data starts and lower on the inside tracks where data ends but it would not explain why it would slow so drastically to ~1MB/s

– thomasrutter
Jan 14 '18 at 23:50





I was going to talk about how HDD transfer rate is higher at the outside tracks where data starts and lower on the inside tracks where data ends but it would not explain why it would slow so drastically to ~1MB/s

– thomasrutter
Jan 14 '18 at 23:50













Could be related to ext4s delayed allocation. However on its own that shouldn't be causing such a regression.

– jdwolf
Jan 15 '18 at 1:01





Could be related to ext4s delayed allocation. However on its own that shouldn't be causing such a regression.

– jdwolf
Jan 15 '18 at 1:01










1 Answer
1






active

oldest

votes


















0














Unfortunately, this is normal and expected for your big file use case. Your case of two hard disks, and a 50G+ file eliminates a lot of misleading talk of "slow devices", "slow buses", and "slow filesystems", and you are left with the unexplained problem of a slow copy. You must have quite a bit of memory to get the performance you have for 30G files. System buffers are used, filled up, and after your copy command finishes, eventually will get flushed to the target, making real timing/rates somewhat difficult (even the "time" command will finish long before the buffers finally get flushed.



The only "workaround" I have found is to use a "copy" command which allows you to set up explicit buffers yourself, like tar or cpio can do. Setting a 2M buffer on tar allowed me to speed up a 10M/sec copy of a 50G file to about 35M/sec -- still much slower than the nominal 100M/sec I get on smaller files (or in Windows).






share|improve this answer


















  • 1





    Thanx for your reply. In the meantime I have tried some additional test, and what I observe is in line with your answer.

    – user3074126
    Jan 16 '18 at 14:22










Your Answer








StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "89"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













draft saved

draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2faskubuntu.com%2fquestions%2f995946%2fdecrease-of-transfer-rate-when-copying-large-amount-of-data%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









0














Unfortunately, this is normal and expected for your big file use case. Your case of two hard disks, and a 50G+ file eliminates a lot of misleading talk of "slow devices", "slow buses", and "slow filesystems", and you are left with the unexplained problem of a slow copy. You must have quite a bit of memory to get the performance you have for 30G files. System buffers are used, filled up, and after your copy command finishes, eventually will get flushed to the target, making real timing/rates somewhat difficult (even the "time" command will finish long before the buffers finally get flushed.



The only "workaround" I have found is to use a "copy" command which allows you to set up explicit buffers yourself, like tar or cpio can do. Setting a 2M buffer on tar allowed me to speed up a 10M/sec copy of a 50G file to about 35M/sec -- still much slower than the nominal 100M/sec I get on smaller files (or in Windows).






share|improve this answer


















  • 1





    Thanx for your reply. In the meantime I have tried some additional test, and what I observe is in line with your answer.

    – user3074126
    Jan 16 '18 at 14:22















0














Unfortunately, this is normal and expected for your big file use case. Your case of two hard disks, and a 50G+ file eliminates a lot of misleading talk of "slow devices", "slow buses", and "slow filesystems", and you are left with the unexplained problem of a slow copy. You must have quite a bit of memory to get the performance you have for 30G files. System buffers are used, filled up, and after your copy command finishes, eventually will get flushed to the target, making real timing/rates somewhat difficult (even the "time" command will finish long before the buffers finally get flushed.



The only "workaround" I have found is to use a "copy" command which allows you to set up explicit buffers yourself, like tar or cpio can do. Setting a 2M buffer on tar allowed me to speed up a 10M/sec copy of a 50G file to about 35M/sec -- still much slower than the nominal 100M/sec I get on smaller files (or in Windows).






share|improve this answer


















  • 1





    Thanx for your reply. In the meantime I have tried some additional test, and what I observe is in line with your answer.

    – user3074126
    Jan 16 '18 at 14:22













0












0








0







Unfortunately, this is normal and expected for your big file use case. Your case of two hard disks, and a 50G+ file eliminates a lot of misleading talk of "slow devices", "slow buses", and "slow filesystems", and you are left with the unexplained problem of a slow copy. You must have quite a bit of memory to get the performance you have for 30G files. System buffers are used, filled up, and after your copy command finishes, eventually will get flushed to the target, making real timing/rates somewhat difficult (even the "time" command will finish long before the buffers finally get flushed.



The only "workaround" I have found is to use a "copy" command which allows you to set up explicit buffers yourself, like tar or cpio can do. Setting a 2M buffer on tar allowed me to speed up a 10M/sec copy of a 50G file to about 35M/sec -- still much slower than the nominal 100M/sec I get on smaller files (or in Windows).






share|improve this answer













Unfortunately, this is normal and expected for your big file use case. Your case of two hard disks, and a 50G+ file eliminates a lot of misleading talk of "slow devices", "slow buses", and "slow filesystems", and you are left with the unexplained problem of a slow copy. You must have quite a bit of memory to get the performance you have for 30G files. System buffers are used, filled up, and after your copy command finishes, eventually will get flushed to the target, making real timing/rates somewhat difficult (even the "time" command will finish long before the buffers finally get flushed.



The only "workaround" I have found is to use a "copy" command which allows you to set up explicit buffers yourself, like tar or cpio can do. Setting a 2M buffer on tar allowed me to speed up a 10M/sec copy of a 50G file to about 35M/sec -- still much slower than the nominal 100M/sec I get on smaller files (or in Windows).







share|improve this answer












share|improve this answer



share|improve this answer










answered Jan 15 '18 at 5:20









ubfan1ubfan1

9,74941730




9,74941730







  • 1





    Thanx for your reply. In the meantime I have tried some additional test, and what I observe is in line with your answer.

    – user3074126
    Jan 16 '18 at 14:22












  • 1





    Thanx for your reply. In the meantime I have tried some additional test, and what I observe is in line with your answer.

    – user3074126
    Jan 16 '18 at 14:22







1




1





Thanx for your reply. In the meantime I have tried some additional test, and what I observe is in line with your answer.

– user3074126
Jan 16 '18 at 14:22





Thanx for your reply. In the meantime I have tried some additional test, and what I observe is in line with your answer.

– user3074126
Jan 16 '18 at 14:22

















draft saved

draft discarded
















































Thanks for contributing an answer to Ask Ubuntu!


  • Please be sure to answer the question. Provide details and share your research!

But avoid


  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2faskubuntu.com%2fquestions%2f995946%2fdecrease-of-transfer-rate-when-copying-large-amount-of-data%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Möglingen Índice Localización Historia Demografía Referencias Enlaces externos Menú de navegación48°53′18″N 9°07′45″E / 48.888333333333, 9.129166666666748°53′18″N 9°07′45″E / 48.888333333333, 9.1291666666667Sitio web oficial Mapa de Möglingen«Gemeinden in Deutschland nach Fläche, Bevölkerung und Postleitzahl am 30.09.2016»Möglingen

Virtualbox - Configuration error: Querying “UUID” failed (VERR_CFGM_VALUE_NOT_FOUND)“VERR_SUPLIB_WORLD_WRITABLE” error when trying to installing OS in virtualboxVirtual Box Kernel errorFailed to open a seesion for the virtual machineFailed to open a session for the virtual machineUbuntu 14.04 LTS Virtualbox errorcan't use VM VirtualBoxusing virtualboxI can't run Linux-64 Bit on VirtualBoxUnable to insert the virtual optical disk (VBoxguestaddition) in virtual machine for ubuntu server in win 10VirtuaBox in Ubuntu 18.04 Issues with Win10.ISO Installation

Torre de la Isleta Índice Véase también Referencias Bibliografía Enlaces externos Menú de navegación38°25′58″N 0°23′02″O / 38.43277778, -0.3838888938°25′58″N 0°23′02″O / 38.43277778, -0.38388889Torre de la Illeta de l’Horta o Torre Saleta. Base de datos de bienes inmuebles. Patrimonio Cultural. Secretaría de Estado de CulturaFicha BIC Torre de la Illeta de l’Horta. Dirección General de Patrimonio Cultural. Generalitat ValencianaLugares de interés. Ayuntamiento del CampelloTorre de la Isleta en CastillosNet.org